Google, IBM & Meta Certificates – 40% Off
One plan covers every Professional Certificate on Coursera.
Unlock All Certificates
Explore an innovative adversarial defense strategy against unauthorized web crawling in this 23-minute conference talk from USENIX Security '25. Learn how to combat increasingly aggressive data collection by large language model companies that ignore or loosely interpret the Robots Exclusion Standard (robots.txt) when scraping web content. Discover Venom, an experimental toolkit that combines advanced fingerprinting and inline proxy techniques to dynamically serve different or misleading content to identified crawlers based on request headers, behavior patterns, and known crawler infrastructure. Examine the practical implementation challenges, legal and ethical considerations, and effectiveness against both text-based and image-based crawling strategies. Analyze case studies demonstrating how LLMs trained on intentionally "poisoned" content experience degraded performance, making large-scale crawling counterproductive for data harvesters. Understand how this approach differs from traditional blocking methods or CAPTCHA solutions by fundamentally reshaping the cost-benefit equation to make unscrupulous data collection efforts yield poor results.
