Discussions
robots.txt
Posted by wakish • 5/11/07 • Subscribe to this Discussion [RSS] • Report This Topic
Topics: consensus, crawler, protocol, robots exclusion protocol, robots exclusion standard, robots.txt, search engine, web browser, web robots, web spiders
I have just written an article on "robots.txt - why is this simple file still so widely used?"
Please view it at: wakish.info/?p=27
Let me know your views..
Thanks!
Add Your Comment
Login to leave a message.


