Discussions
Why is blogger blocking my blog from being indexed??
Posted by Mkcoy • 7/09/09 • Subscribe to this Discussion [RSS] • Report This Topic
Topics: blogger, Help
Hi I was just working on submitting my site to some search engines i was using scrubtheweb.com.
Anyway after submission I get this message:
===================================
Warning! The robots.txt file located at how-to-earn-swagbucks.blogspot.com/robots.txt blocks the submitted URL from being indexed by the robot(s) with the name of:
* ALL robots blocked from indexing this URL!
If this is your Web page and you do not wish to restrict the above robot(s) from visiting this page, it is recommended you make the necessary changes before submitting.
===================================
How can I change the robots.txt at how-to-earn-swagbucks.blogspot.com/robots.txt ?
Can it be done? Why is it there :S
User Comments
-
First, go to the settings page on your blog and make sure it is searchable.
Your blog may also have a security feature that doesn't allow bots. I don't know how to enable/disable or circumvent that, but you may have to address that issue. -
-
Ok so I looked at your robots.txt and there is nothing abnormal about it. The search directory doesn't matter, it just contains your labels, and either way, you can't change that, and they won't change it for you server side either. Search engines usually use your sitemap so they aren't going to miss anything by not having the /search directory.
-
-
@ trailofpen. It's apparantly blocking any search engine from indexing my site.
I was recieving steadily climbing visitors from Google and then all of a sudden they just stopped. And have not had any hits from Google search engine traffic now for about 5 days it's weird.-
If you submit your sitemap to search engines, then you won't have any problems. The /search directory does no affect your pages being indexed or not. I have blogger sitemaps submitted to google and the robots.txt has never given me problems even though it looks almost the same as yours. /search only contains your labels, nothing more.
-
@MCoy
Are you running Google Adsense ads on your blog?I'm asking because a quick Google search does turn up problems that sound like yours that were happening in 2008 tinyurl.com/create.php
Also have you contacted Blogger staff and requested and explanation and/or assistance? That would be the first thing that I would do if this was happening with my blog.
-
@ dcaroll No I've not violated any TOS or Blogger or Adsense TOS. It's nothing to do with that. It was just an error notice I got when submitting my blog to scrubtheweb.com. Not sure why it said "ALL robots blocked from indexing this URL!" but now I think I know that it doesn't matter. It's just a error notice that, that site gives me based on its own settings on search engines as it knows it. After looking up on it and asking on the blogger help forum here. www.google.com/support/forum/p/blogger/thread?tid=316dbea2d94013e5&hl=en
timethief yes I am and have seeked help via the Google forum as above thankyou Also the link you provided didn't load.
trailofpen Yeah I don't think now after looking it up it's anything to worry about. Would like to know more about the cse.xml and what it actually does but think I can submit without worry. It was probably just what that site said for whatever reason making me think that was the reason why the sudden drop in organic traffic. -
dcarroll thinking about it. Don't know why I didn't think of it. My blog was recently flagged and locked due to possible Blogger Terms of Service violations. But I requested an unlock review. I got to do captcha everytime I post. Also tweetmeme is having some problems crawling it as well help.tweetmeme.com/2009/04/15/button-faq/#2
that could be because the crawler timed out itself though.
But some images are not loading on my blog and even feedburner is not displaying the chicklet right. Are all these things together a coincedence? Could it be why I'm not getting the same organic traffic results anymore? What do you mean by sanctions?
Edit other search engine traffic is picking up actually Yahoo MSN etc just not the big G. No love for me
Wish i knew how to crack it.
Does the top right banner display? how-to-earn-swagbucks.blogspot.com/
Add Your Comment
Login to leave a message.



