Screen Scraper Tricks Extracting Data from Difficult Websites
Michael Schrenk explores screen scraper tricks extracting data from difficult websites at DEF CON 17, examining electronic-locks and practical defenses.
Summary
Architecture and Background
Screen Scraper Tricks: Extracting Data from Difficult Websites Michael Schrenk Screen scrapers and data mining bots often encounter problems when extracting data from modern websites. Obstacles like AJAX discourage many bot writers from completing screen scraping projects. The good news is that you can overcome most challenges if you learn a few tricks.
Vulnerability Analysis
This session describes the (sometimes mind numbing) roadblocks that can come between you and your ability to apply a screen scraper to a website. You'll discover simple techniques for extracting data from websites that freely employ DHTML, AJAX, complex cookie management as well as other techniques. Additionally, you will also learn how "agencies" create large scale CAPTCHA solutions.
Exploitation and Demonstrations
All the tools discussed in this talk are available for free, offer complete customization and run on multiple platforms. Michael Schrenk is a webbot developer and the author of "Webbots, Spiders, and Screen Scrapers" (2007, No Starch Press). He has also written for ComputerWorld, php|architect and Web Techniques magazines.
Remediation and Defensive Strategies
Mike also gave presentations at DEFCON X, XI and XV. He works for a wide range of clients across North American as well as in Russia, Spain and The Netherlands. Stop by www.schrenk.com and say hello. For copies of the slides and additional materials please see the DEF CON 17 Archive here:
Related talks
Main Stage·12 min
Ricky Lawshae
Ricky Lawshae explores picking electronic locks using tcp sequence prediction at DEF CON 17, examining electronic-locks and practical defenses.
Main Stage·46 min
Sho Ho
Sho Ho explores foe feeding controversial news to censored countries at DEF CON 17, examining electronic-locks and practical defenses.
Main Stage·22 min
Pratap Prabhu · Yingbo Song
Pratap Prabhu and Yingbo Song explores smashing the stack with hydra at DEF CON 17, examining electronic-locks and practical defenses.