Comments on: Robot.txt Revisited http://www.cssOrigins.com/2009/robottxt-revisited/ Css Origins, Design Blog and Wordpress Publishing Platform Tue, 06 Jul 2010 21:35:00 +0000 hourly 1 http://wordpress.org/?v=abc By: Hunter Brelsford http://www.cssOrigins.com/2009/robottxt-revisited/comment-page-1/#comment-8 Hunter Brelsford Wed, 08 Apr 2009 21:16:20 +0000 http://www.cssOrigins.com/?p=140#comment-8 Many of the files from my research at least proved that the files can be indexed by bots, and you may not want them too. Search bots out there don't only index links, but rather transverse the web though various links allowed by the site. For instance if you had an include file to a css stylesheet on your site but no actual link to it eventually Google and the other searches would catch on to that and index the contents of your css file. On the other hand if a user installed their blog into a different directory then their root (ie /) like I have the robots.txt file would change accordingly My whole deal was trying to customize the Robots.txt file to get the optimal content indexed by googly without repeating content and thus possibly losing potential natural rankings. thanks for the comment. Many of the files from my research at least proved that the files can be indexed by bots, and you may not want them too.
Search bots out there don’t only index links, but rather transverse the web though various links allowed by the site. For instance if you had an include file to a css stylesheet on your site but no actual link to it eventually Google and the other searches would catch on to that and index the contents of your css file. On the other hand if a user installed their blog into a different directory then their root (ie /) like I have the robots.txt file would change accordingly

My whole deal was trying to customize the Robots.txt file to get the optimal content indexed by googly without repeating content and thus possibly losing potential natural rankings.
thanks for the comment.

]]> By: Jonathan Dingman http://www.cssOrigins.com/2009/robottxt-revisited/comment-page-1/#comment-7 Jonathan Dingman Wed, 08 Apr 2009 15:46:24 +0000 http://www.cssOrigins.com/?p=140#comment-7 Not really sure how much this actually helps since search engines won't access those files in the first place unless they are linked to. And be careful when doing that, some people may have WordPress installed at /wp/ in which case, would essentially block their site from the search engines. Not really sure how much this actually helps since search engines won’t access those files in the first place unless they are linked to.

And be careful when doing that, some people may have WordPress installed at /wp/ in which case, would essentially block their site from the search engines.

]]>