Inside the HostGator cpanel you have a tool called “fantastico” which is an automated script installed. It installs a variety of scripts like WordPress, Joomla, ETC.

Apparently someone thought it would be a good idea to have the robots.txt file set to not allow for indexing of the blog. I have about 20 new websites that this issue created a problem for. Like most people in SEO they know there is a waiting period or “sandboxing” that you need to wait for before Google will start sending traffic and indexing your site nicely. I wanted about 6times longer than it usually takes and then I started getting angry. Why the hell weren’t my sites getting indexed?

The robots.txt was the last place I actually looked, I even checked .htaccess before robots.txt because I knew that I would never EVER set the robots.txt to disallow searc h engines. In fact, the only time I ever create a robots.txt is when I want to black a certain part of a site from being indexed, otherwise I usually don’t even create a robots.txt.

So anyway, to make a long story short I realized that every new blog I had installed using HostGators fantastico over the past 4-6 months had the robots.txt set to not allow indexing.

What’s bad…and embarrassing, is that I’m in the field of web design and SEO and it took my that long to realize what was going on. The main reason it took so long is because I have so many sites I wasn’t really focusing on these wordpress blogs and everytime the question on why they weren’t being indexed came up in my head I would just write it off and sandboxing and tell myself “meh, give it another few weeks”. This ultimately lead to month after month of not being indexed. Finally when I actually stopped what I was doing and really looked into the situation I figured it out. But for the average Joe out there this could have been a serious problem.

I’m sure there are a LOT of people having the same issue as I was having. Some are trying to figure it out but can’t, some probably don’t even know that there is a problem, others don’t even know what a robots.txt is. Hopefully HostGator will look into this issue.

HostGator has all their fantastico WordPress installs set to not allow robots to index the blogs is a post from: Fort Myers Web Design