An Important SEO Component You May Be Missing: Googlebot Optimization

googlebot optimization.png

There is a part of the SEO process that isn't discussed quite as much as the other aspects – the Googlebot, or how Google scans sites for information used in page ranking. And while we love to talk about the value of content (important!) and tags (also important!), it's good to remember that this busy little bot needs all the help that it can get.

That's why Googlebot optimization is a small but important part of your web design, and definitely worth talking about. Here's what to know to make sure Google has no problems with your site.

Spare a Thought for the Bot

Googlebot is constantly going through all sites that it has access to and scanning their content. The Googlebot is very efficient: Every site on the Google search engine is probably scanned once every several seconds, which means constant updates for everyone. Some sites, of course, block Google from scanning them and therefore don't show up on the search engine at all (the dark web is encrypted for a reason), but for normal commercial activities, your website is as familiar with bots as a dog is with fleas. To understand the process, there are three things you need to know about the Googlebot:

  1. Page Ranking. The better the page ranking, the more often the bot crawls the site. The bot doesn't waste time with very low-ranked sites, but highly ranked sites tend to both change more often and see more traffic, so the bot scans those more frequently. The difference may only be a few seconds, but it adds up over time.
  2. Robot.txt. The bot looks at the "robot.txt" file to see where to crawl. This particular file tells the bot what it can and cannot scan. For older sites or more complex sites, robot.txt is important because it's essentially a list of pages that Google will ignore, including especially shoddy pages that need work, pages with customer data that's very private, and pages that are necessary but have very little information on them and could skew your ranking.  
  3. Crawl Budget. There is a "crawl budget" for your site. Combine the first two points, and the Googlebot creates an allotment for just how long it's going to spend on your site. Generally, a bigger crawl budget is better for your page ranking, or at least indicative that you have a healthy site. Yoast has more information on how to figure out your own crawl budget and we suggest taking a look.

Optimization: New Content Is Important

So, how do you optimize for the Googlebot? Don't worry – it's a simpler task than most optimization, but it's necessary to consider several important aspects. The first is new content. The Googlebot and search engine both love to see regular, new content being posted to a website in the form of blogs, updates, and so on. It shows the site is active, encourages more traffic, and generally helps the bot favor you. However, if the content isn't high quality then the bot may not like what it sees, so focus on strong content to avoid losing page rankings.

Don't Confuse the Bot

The Googlebot is learning, but it still struggles with some more complicated web design. It doesn't handle Flash or DHTML very well, it still struggles with JavaScript and Ajax, and it can get confused by more complex HTML setups. If you have animations, infinite scrolling, or other features on your site, make them as simple as possible and avoid more complex scripts when possible. This is also great for mobile optimization!

Use Internal Links for Your Best Content

Throw up internal links on your strong content. The Googlebot likes these links because it makes it easier to scan the full site and get an idea of what is where and what your content is about.

Use the Right Tools

Newer SEO tools like Botify have the ability to mimic the Googlebot and scan your site to see what the bot sees and provide advice on optimization. If you can find a tool like this to help out, why not use it?

Download SEO Audit Checklist

Aleks Vizulis

As operations manager at 21 Handshake, I bring an analytical perspective to the discussion table. My vision is to take projects from 'just strong' to superb. Often you will find me capturing behind the scenes photos in the 21 Handshake office or at a clients location. I play a fierce round of 18 during warm weather and catch me on the slopes during the other months of the year!

Find me on: