Recent Posts Tagged With 'webmaster central'

• Debugging robots.txt with Google Webmaster Tools

  Posted on Tuesday August 18th, 2009 at 12:32 in google, SEO, webmaster central, robots.txt, redirects, crawler directives

  Although Google’s Webmaster Console is a really neat toolkit, it can mislead the not-that-savvy crowd every once in a while. When you go to “Diagnostics::Crawl Errors::Restricted by robots.txt” and you find URIs that aren’t di...

• Getting URLs outta Google - the good, the popular, and the definitive way

  Posted on Monday January 14th, 2008 at 15:01 in web development, duplicate content, webmaster central, robots.txt, crawler directives, robots meta tags, x-robots-tag, url removal

  There’s more and more robots.txt talk in the SEOsphere lately. That’s a good thing in my opinion, because the good old robots.txt’s power is underestimated. Unfortunately it’s quite often misused or even abused too, usually be...

• Validate your robots.txt - Googlebot becomes smarter

  Posted on Tuesday November 20th, 2007 at 12:58 in web development, recommendations, testing, webmaster central, robots.txt, crawler directives

  Last week I reported that Google experiments with new crawler directives for use in robots.txt. Today Google has confirmed that Googlebot understands experimental REP syntax like Noindex:. That means that forgotten –and, until recently, ignored...

• Q&A: An undocumented robots.txt crawler directive from Google

  Posted on Wednesday November 14th, 2007 at 11:05 in web development, webmaster central, robots.txt, crawler directives

  Blogging should be fun every now and then. Today I don’t tell you anything new about Google’s secret experiments with the robots exclusion protocol. I ask you instead, because I’m sure you know your stuff. Unfortunately, the Q&...