There is a part of the SEO process that isn't discussed quite as much as the other aspects – the Googlebot, or how Google scans sites for information used in page ranking. And while we love to talk about the value of content (important!) and tags (also important!), it's good to remember that this busy little bot needs all the help that it can get.
That's why Googlebot optimization is a small but important part of your web design, and definitely worth talking about. Here's what to know to make sure Google has no problems with your site.
Spare a Thought for the Bot
Googlebot is constantly going through all sites that it has access to and scanning their content. The Googlebot is very efficient: Every site on the Google search engine is probably scanned once every several seconds, which means constant updates for everyone. Some sites, of course, block Google from scanning them and therefore don't show up on the search engine at all (the dark web is encrypted for a reason), but for normal commercial activities, your website is as familiar with bots as a dog is with fleas. To understand the process, there are three things you need to know about the Googlebot:
- Page Ranking. The better the page ranking, the more often the bot crawls the site. The bot doesn't waste time with very low-ranked sites, but highly ranked sites tend to both change more often and see more traffic, so the bot scans those more frequently. The difference may only be a few seconds, but it adds up over time.
- Robot.txt. The bot looks at the "robot.txt" file to see where to crawl. This particular file tells the bot what it can and cannot scan. For older sites or more complex sites, robot.txt is important because it's essentially a list of pages that Google will ignore, including especially shoddy pages that need work, pages with customer data that's very private, and pages that are necessary but have very little information on them and could skew your ranking.
- Crawl Budget. There is a "crawl budget" for your site. Combine the first two points, and the Googlebot creates an allotment for just how long it's going to spend on your site. Generally, a bigger crawl budget is better for your page ranking, or at least indicative that you have a healthy site. Yoast has more information on how to figure out your own crawl budget and we suggest taking a look.
Optimization: New Content Is Important
So, how do you optimize for the Googlebot? Don't worry – it's a simpler task than most optimization, but it's necessary to consider several important aspects. The first is new content. The Googlebot and search engine both love to see regular, new content being posted to a website in the form of blogs, updates, and so on. It shows the site is active, encourages more traffic, and generally helps the bot favor you. However, if the content isn't high quality then the bot may not like what it sees, so focus on strong content to avoid losing page rankings.
Don't Confuse the Bot
Use Internal Links for Your Best Content
Throw up internal links on your strong content. The Googlebot likes these links because it makes it easier to scan the full site and get an idea of what is where and what your content is about.
Use the Right Tools
Newer SEO tools like Botify have the ability to mimic the Googlebot and scan your site to see what the bot sees and provide advice on optimization. If you can find a tool like this to help out, why not use it?