Discussions

Confused about URLs restricted by robots.txt

Posted by nplbjirel • 3/24/08 • Subscribe to this Discussion [RSS] • Report This Topic
Topics: blog, google, label, tag

When I checked my blog report in Google Webmaster tool just a while ago, I found that 63 URls were restricted by robot.txt- all of them the Label link.

Do you know what caused this? That means it'd be better if I remove the Tag( Label) cloud from by blog?

User Comments

3/24/08

xtremer

do you have a blogger blog?

[reply]
1. 3/24/08
  
  nplbjirel
  
  Yes xtremer
3/24/08

umesh

On blogspot blogs url with /search are restricted by google /blogger to avoid duplicates in search engine index.
There is no need to do anything for this.
If any other post is restricted not having /search you need to find out the reason.

[reply]
1. 3/24/08
  
  nplbjirel
  
  No, they are label link.
3/24/08

umesh

All label links will have /search.
Like this yourblog.blogspot.com/search/label/labelname
They are restricted by robots.txt for the reason mentioned above.

[reply]
1. 3/24/08
  
  nplbjirel
  
  Thanks!
3/24/08

ThriftShopRomantic

Well, that's good to know. I saw that myself and was wondering.

Glad you asked Nplbjirel!

[reply]
4/09/09

wagerwitch

OK - so they are coming up as errors - they never did before - so does this means I should ignore them?

It's scary to see ERRORS
LOL

[reply]
4/25/09

gagirliegirl

I just started seeing this too... suddenly 3 of my labels had this error URLs restricted by robots.txt errors .... I don't see why they suddenly started erroring... and why only these three labels and not the others... I have tons of labels....

Somehow the explanation given doesn't make sense... why some and not others and why all of a sudden...

[reply]
6/16/09

toriola1

I am facing the same thing with my blog: online-travel-agent-guide.blogspot.com and I am surprised that some URL's have the error while others do not have the error. One confusing thing is that all these URL's have the same structure. Does anyone know the reason for this "URLs restricted by robots.txt errors"?

[reply]
6/17/09

umesh

you can check your robots.txt file at yourblog.blogspot.com/robots.txt
You can find disallow :/search there.
This means all url having /search within them will be restricted.And hence we get error as url restricted by.If all labels ( label has /search in url) does not get this error means it is search engine crawling problem.If any url not having /search is restricted then you need to worry and try to find out the reason.

[reply]
1. 7/16/09
  
  tanzy
  
  Anyone who can help us fix this issue. Does anyone know how to get rid of disallow: /search by taking some steps ..
  
  Anything would be helpful.
  
  Thanks.
7/16/09

husdal

As umesh said, there is a very good reason for it to be so and there is no reason to want to fix this. It's actually better for your search results in Google if Google does not index the label pages. Non-blogspot blogs have to write their own robots.txt file manually, otherwise their category and tag pages will be indexed. Anyway, it's perfectly good SEO to do it, so there is no reason to be alarmed. What is interesting, though, is that Gogle apparently started doing this by itself for blogspot blogs.

[reply]
7/16/09

gagirliegirl

This is okay.... no need to do anything.... they are restricted because they are labels and labels contain posts that are already crawled.... this avoids unnecessary duplicates... you only need to do something if something other than a label is restricted...

I had this start around April or May... I questioned it then and this was the explanation I received from Google... my guess is they are cleaning up that's why it wasn't done before.

[reply]
9/04/09

jollyjcb

comment removed by the community.

[reply]