Before

Completely unsure about how good bots scan and index files and sites, myself and other bloggers alike had come up with a million different ways to present their own robots.txt file, and a million more ways to do theirs if they where using a wordpress site.

they ranged from

User-agent:  *
Disallow: /
Disallow: *

to something similar to this.

User-agent:  *
Disallow: /wp*

After

Some of what we have in our typical robots.txt is indeed valid, but bear with me for a second as i explain what i did to come up with some new results.

to gather statistics on what was being indexed I cross referenced the results from some analytic tools that I had available.

1. Google analytics (google search stats)
   
2. Stat press (wordpress stats)
   
3. awstats (server side built in stats)
4. Web mastering tools (By Google)
   

From the data I was able to pull from these resources I specifically looked for what was being hit. Because of the special way that wordpress loads, and imports I could see that /index.php was being indexed as a file, but most other FILES on the site where not indexed. All other PAGES or what would be considered a page by a typical computer user where not indexed as an actual FILE but rather indexed by its permalink location.

hmm … interesting Google bot indexes files and paths. Interesting  but not unexpected.

By now most of us know this, you set up a permalink structure in your website and it dynamically changes the links throughout your site to reflect the change. This is great for dynamically driven websites looking for a cleaner path or some SEO optimization.

The Google bot can in most cases follow an average custom permalink structure, I will note that the more difficult the permalink structures are ether crawled slower if crawled at all. I would shy away from lots of string names within your structure =?etc …

Back to robots…

So how does all this effect your robots.txt structure you ask? Well if you have gotten yourself a good permalink structure and you can then you can essentially set up your robots to block everything but the categories, pages, or posts you want it to crawl, and is perfect for helping eliminate duplicated content.

Here’s an example.

My own robots.txt file

User-agent:  *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /category/archive/
Disallow: /feed/

I have disallowed my wp-admin directory, and a category in my wordpress permalink structure /category/archive, and finally disallowed my rss feeds, accessed through http://www.cssorigins.com/feed if my site was indexed properly by Google then all the content that would be in the feeds would already been indexed and thus the feeds would be duplicated content. The same goes for the archive category, since I post all my posts into that category as well as their individual categories.

But remember that the bots index files as well, so it would be a bad idea to do something similar to this.

Disallow: /*.php

as your index file now would not be indexed, along with the rest of your website. =(

Don’t understand? Think there’s a better way? Please comment and help us build a better understanding of how the web works.

No related posts.

Anyways, through development of my own site I’ve come up with a good list of plug-ins and information that could be useful when optimizing your own website.

1. SEO optimization plug-in for wordpress.

there are lots of plug-ins out there even in the seo field, I currently have been using the “ALL IN ONE SEO ” plug-in produced by semper fi web design . Highlights include

• automatically optimizes your titles for search engines
• Generates META tags automatically
• Avoids the typical duplicate content found on WordPress blogs
• For WordPress 2.7.1 you don’t even have to look at the options, it works out-of-the-box. Just install.
• You can override any title and set any META description and any META keywords you want.
• You can fine-tune everything
• Backward-Compatibility with many other plug-ins, like Auto Meta, Ultimate Tag Warrior and others.

2. robots.txt

Your robots.txt file is important to your website and your seo optimization specifically with a wordpress site because it will disallow indexing duplicate content in your website or blog. Most people know that links back and forth in your site and to other sites help bring up your natural listings in search engines and they do but duplicate content can actually hurt you as well.  With that being said wordpress has no static pages at all, all pages are generating from a header.php page. So if we only wanted to index the pages that don’t have duplicated content on it what do we index and what don’t we?

That is a very difficult question to answer, unless your extremely familiar with the inner workings of wordpress you may not know what’s what or where to disallow, so let me share my two cents and we will compare and contract with some of what others have said.

My robots.txt can be found here .

User-agent:  *   <-- * means everyone that would try to index my page all bots
Disallow: /cgi-bin/  <-- our first disallow statement, removed this directory from being indexed by *(all)

There are many disallow statements in my robots.txt they all cover different folders or file extensions. But the real controversial ones would be something like this

User-agent:  *
Disallow: /wp-*

That would completely disallow the indexing of all of your wordpress folders. wp-admin, wp-includes, and wp-content. wp-content being where your theme files are save I in my own opinion think that this area should be indexed and loaded along with some if not all of your plug-ins. My reason for believing this is because these pages are called with content attached to them when the page is generated originally. All of your files in the theme folder I consider to be as important to seo as any normal websites pages would be to it. Please someone enlighten me.

3. SEO friendly permalink structure

The wordpress default permalink structure looks something like this { http://www.yourwebsite.com/?p=123 ie:UGLY } when you publish a new page or post you get returned a string query for your post name. It’s well coded and it works but it’s

1. not easy to remember
2. not seo friendly
3. on average makes several more calls to your database then required

permalink structures in wordpress are completely designable based on the criteria that your wanting to get across, for larger sites with lots of extra pages i would try and start your structure with a number like /%year% the numbers help wordpress distinguish between post URI and page URI and is mainly a performance consideration.

The second basic not to do is to have just /%postname%/ ie { http://www.yourwebsite.com /%postname%/ } in wordpress 2+ it can break some of the functionality and limit your websites ability to access some of the files on site.
I personally will be using this permalink structure. /%year % /%postname%/
more information about the wordpress permalink structure at their codex .

No related posts.