Recent Posts Tagged With 'xml-sitemaps'

• Get a grip on the Robots Exclusion Protocol (REP)

  Posted on Thursday January 17th, 2008 at 08:26 in web development, microformats, robots.txt, crawler directives, robots meta tags, x-robots-tag, xml-sitemaps, url removal

  Thanks to the very nice folks over at SEOmoz I was able to prevent this site from becoming a kind of REP/robots.txt blog. Please consider reading this REP round up: Robots Exclusion Protocol 101 My REP 101  links to the various standards (robots...

• My plea to Google - Please sanitize your REP revamps

  Posted on Thursday January 3rd, 2008 at 18:02 in web development, msn, microformats, nofollow, robots.txt, crawler directives, robots meta tags, x-robots-tag, xml-sitemaps

  Standardization of REP tags as robots.txt directives This draft is kinda request for comments for search engine staff and uber search geeks interested in the progress of Robots Exclusion Protocol (REP) standardization (actually, every search engine m...

• Google to change the Robots Exclusion Protocol again

  Posted on Friday November 30th, 2007 at 12:33 in microformats, nofollow, robots.txt, crawler directives, robots meta tags, x-robots-tag, xml-sitemaps

  Web crawler directives, partly standardized in the Robots Exclusion Protocol (REP), evolved since 1994. Nowadays we’ve to deal with a conglomerate of not binding de facto standards and microformats, all of them extended by various organizations...