Discussions
Urls restricted by robots.txt in webmaster tools for my blogger blog sitemap..
Posted by pixelshots • 5/04/09 • Subscribe to this Discussion [RSS] • Report This Topic
Topics: pixelshots, url's restricted by robots.txt, webmaster tools
I have a problem of urls getting restricted by robots.txt in of my blogger blog www.pixelshots.blogspot.com
I havent edited the robots.txt and by default in the disallowed links section /search is given in default... I have got 179 links restricted in the webmaster tools and is there a way to solve it..
I can't edit the robots.txt in blogger and is there any way to do it..
The urls getting blocked are from the labels given under the posta and all the url's having /search in the links...
Do u people have any solution....Is this problem normal with blogger blogs??
User Comments
-
Do not worry, it is for your own good, that the url s are blocked.
Those are the urls of your labels, to prevent duplicate content indexing by google.
To better understand this, consider this example:
You have 2 posts:
1. how to make pineapple pastry. (url 1)
2. how to make chocolate pastry. (url 2)
Now both of them have a common tag pastry. And this tag when you click on it, you'll see url 3.
And this url 3 will have the exact content as in url1+url2
that means Google may think that you copied content from url1 and url2 in to url3, and will penalize url3, which is nothing but your own website(actually all 3 urls are your own)
So if we stop Google from indexing url3, then there won't be any problem of content duplication.
ONE MORE PROBLEM which HAS NOT BEEN taken care of, is the content duplication at "Blog Archives", because if someone see your articles (of say oct 2008/jan 2007) it will show articles of the complete month with a different url.
To avoid this issue, I advice you to hide articles and only show titles in archives (bloggerstop.net/2009/01/show-only-titles-in-blog-archives-with.html).
Regards
Sai
Add Your Comment
Login to leave a message.

