Discussions

Discussions » Shameless Blog Promotion
wakish

robots.txt

Posted by wakish • 5/11/07 • Subscribe to this Discussion [RSS] • Report This Topic
Topics: consensus, crawler, protocol, robots exclusion protocol, robots exclusion standard, robots.txt, search engine, web browser, web robots, web spiders

I have just written an article on "robots.txt - why is this simple file still so widely used?"
Please view it at: wakish.info/?p=27

Let me know your views..
Thanks!

Reply

User Comments

  1. gadegigo0
    5/11/07
    gadegigo0
    Hmmm, don't think i've seen this file before
    [reply]
  2. wakish
    5/11/07
    wakish
    Then have a look at my article
    [reply]
  3. bladeaxe4
    1/31/09
    bladeaxe4
    digged it !
    [reply]

Add Your Comment

Login to leave a message.