Script Installation
Robots.txt parser Script Installation and Configure Services
|
Script Description:Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details). Site Url: http://hotscripts.com/jump.php?listing_id=53640&jump_type=0 |