Troubleshooting Linking and Crawling Errors

Lesson Details:

I: Introduction

There is a lot of content available on the web and the search engines index and rank only a small part of it. This is true for most search engines, and is even more apparent on Google.

Google wants to bring the best and most relevant content to its users, so it uses several kinds of factors, such as:

– PageRank – number of quality incoming links to your site (and not quantity as some people think),

– Authority – how authoritative is your domain,

– Crawling errors – an error encountered by Google’s crawlers that prevents them from indexing your page correctly.

The impact of those factors is highly debated and depends on the link and topic and with what domain you are working. But, in general terms, we can say that:

– The more quality links you have, the more authority will be attributed to you;

– The less errors Google crawlers encounter on your site, the more authority you will have.

In order to have a successful SEO strategy, then, you need to work on building quality links and reducing crawling errors. In this article I will focus on the latter, but will mention the former as well.

II: Body

A: Troubleshooting linking and crawling errors

Google encourages webmasters to use a tool called Fetch as Google to diagnose any linking or crawling issues they may be having with their site. You can use it by going to: https://fetchasgoogle.com/ . It is a very helpful tool to see if there are any linking or crawling issues. If you don’t know how to use it, you can find a great tutorial here: http://www.seomoz.org/blog/fix-404s-by-fetching-as-google.

One of the most common issues is the “link loop” issue. This occurs when a page links to itself or another domain on a page using a “rel=” tag on a link. When Google sees a page with a link on it pointing directly to itself, it assumes that the page is not very good and does not want it indexed. To fix the link loop issue, you should go to Webmaster tools (https://www2.google.com/webmasters/hacking-search-engines/) , click on “Crawl Errors” and then select “Fixing Linking Issues”. The next step is to delete the page that is linking to itself using the “Delete URL” button. After that, it is necessary to review all the other pages that were linked to by the deleted page and do the same thing to them. For example, if page A links to B, B links to C and C links back to B, then you would need to delete page B and delete all the links back to it from C. This way you make sure that you clean up all your links at once. To do this go to Google Webmaster Tools (https://www2.google.com/webmasters/hacking-search-engines/) , search for the page that you want to remove using the “Search” button over there and then click on “Remove URLs” next to it. Also try deindexing the domains that are pointing back to each other using the “Deindex Domains” button above all of them. This way you will be sure that your site is not being penalized for duplicate content.

B: Troubleshooting blocking issues

Another issue that might happen frequently is when Google blocks your site because it thinks that something has changed since it last crawled it, so it doesn’t trust your site anymore. If this happens, then go back to Google Webmaster Tools (https://www2.google.com/webmasters/hacking-search-engines/) , click on “Crawl Errors” and then select “Fixing Blocking Issues” to see if there are any changes that might have caused this problem. If there are, make sure you fixed them immediately because this could cause your site to be penalized by Google or even blocked by it altogether.