Discussions
Robots.txt - Do you have one? or have questions?
Posted by werelax • 9/21/08 • Subscribe to this Discussion [RSS] • Report This Topic
Topics: duplicate content, robots.txt
Hi, I use robots.txt on all of my blogs and my website. Primarily to restict spidering of my admin sections and to reduce duplicate pages.
On my blogs I want people that come to my blog by search to end up on category pages, product pages, or individual post pages. So I block tagged, archive and search pages.
You can see one of them here. www.painrelief101.com/robots.txt since all of my blogs are on wordpress I use pretty much the same robots.txt on each blog.
User Comments
-
Ditto
I think in the long run they help you achieve a slightly higher page rank too, don't they? Or at least block Google bots from indexing unimportant pages? -
HERE'S MINE .. www.petlvr.com/blog/robots.txt
>> but, I didn't create it. if you add the All-In-One-SEO plugin to your wordpress blog .. go to settings, and about halfway in there's a box to tick to include the creation of the robots.txt file. -
@annaswan - In some cases they will help significantly in improving Google search results by stopping indexing of duplicate pages (a real problem with most blog software.
-
Ahhh I thought so But a web developer I am not. I learned basic html, a little java - and then threw my hands up and decided templates rule!
-
Add Your Comment
Login to leave a message.
