Getting the Most Out of Google Search Console’s Coverage Reports

I am Nefise, one of the team members of “SEO Painkillers”. I am here to share a summary of our latest Weekly SEO of March 18, 2022.

In this episode of SEO Weekly, Ohio-based Technical SEO strategist Brenda Malone delves into the Coverage Report in Google Search Console.

• Host: Roman Adamita, Director of SEO at BoostROAS
• Guest Speaker: Breanda Malone, Technical SEO strategist

Leveraging The Index Coverage Report to Improve SEO Performance 

We’re going to cover today the Coverage report, which is a specific section of the Search Console.

“The covers”. It’s the second most valuable item in the Search Console. And that’s where I spend most of my time. Most of the analysts at our agency are concerned with the statistics and the up and down graphs. But my forte is getting into that coverage report and seeing what Google sees about your website. We’re going to show What Google sees concerning your website. 

We’re going to discuss the best way to use the coverage report;

How to Automate it?

How often should you check the report?

How Can You Register Your Site in Google Search Console?

And the first thing you want to do with your site, a new site, or any site is to make sure that they are in the Google Search Console. And it’s very easy to do. You just navigate search that google.com/search console. You enter your URL, or it’s better to enter the domain name, but some people freak out because you’ve got to mess with your DNS. So don’t worry about that. If you don’t want to do your DNS, just enter your URL and follow the prompts. Google is beneficial.

And if you get stuck, there is an excellent video showing how you get into your Google Search Console, and it’s the official help from Google for anything to do with the coverage report.

Once you have verified your URL, your website and you are in the console. You’ll see that there are many things to do in all the places you can go. However, our destination is the coverage report. That’s where all of the intense down details that Google is finally sharing with us about how they see your website.

Coverage Report; Where We Get To Work

And as you’ll see, it’s got four tabs; an error tab, valid with a warning tab, the valid green tab and the grey excluded tab. Google will show you exactly how many pages from your website, and it recognizes what category it puts it in. 

The first category is The Error, and that’s the crucial problem that you need to fix right away because it probably is a server error, and if you’ve got a server error, Google can’t and will not come back and view your emphasis recite the robot. We’ll just exit and not come back until you’ve fixed it. So if you have any errors, address those immediately,

The second is the gold box. That’s Valid with Warnings. And Google is saying that your page may or may not be indexed because there is a problem, but you need to go and see what their problem is. 

The third one is Valid. This is the ultimate. This is what you want. You want all of your pages that you want to be indexed to appear in the valid column. You’ve done your job. You can go out Friday night and get a couple of drinks. We’d be having young SEOs here if you’re old enough, sorry. 

And the last box is the most informative for me and for what we do. And it’s The Excluded. This is where you want to concentrate your time because Google was telling you, these are the pages that it will not put in the index, and you need to dig under the hood and see why; if you want those pages included, you’ve got a lot of work to do. 

The Error Tab

If your pages appear here, it could be quite a few things. It could be a server error, and you’re going to get a 5XX when the Google bot tries to go there. Server errors can run the gamut. So there’s nothing I can tell you about what specific error, but you need to talk to your domain host, and they can help you with that. 

It could be a redirect error, and redirect errors are interesting is when you’ve redirected your web page, And you’ve forgotten that you already had a redirect on it. Hence, it redirects, and then you put another redirect, and it just goes on and on and on. And Google bot just gets tired and says, oh hell no. And it goes back. It goes away. 

Or you could be blocking it by the robots.txt and your htaccess file. Google Google is so cool. It will actually tell you. You don’t have to guess why you have an error. If you see along the bottom of the screen where the two errors show, it shows you exactly the reason why it’s in the error category. So. It couldn’t be any easier. Once you click, you get more details about which URLs are affected. So you can personally go and visit the back end of your website and see what the problem is. But errors that should be your pun intended red flag to address right away, or else the Google bot is not going to come back and then not going to get indexed or stay indexed. 

The Valid With Warnings Tab

This one’s not too bad. Google is saying we may have put your page in the index. But we don’t know if you really wanted it in the index because we’ve got conflicting advice from you. Your robots.txt file said, don’t put it in, but on your page, we can’t get to the page to have the robot read what you really want to do with that page and compound matters. Other pages on your site have links back to that page. So Google bot is confused. It’s like, do you want us to follow the links to this page? Or do you want to spend not going to this page? So it’s a warning. It’s like, listen, get your stuff together. Either remove the robots or remove the links to the page and figure out. This yellow tab is kind of important, but it’s the lowest man on the totem pole, actually.

The Valid 

The green valid is eureka. You have submitted your URLs in your XML site map. Google has read it, loved it, validated, visited the pages, found the pages were worthy of being. And that’s another thing most people don’t know. Google has never, ever, ever guaranteed to put every page that you create. 

• Submitted and Index:

The index, it’s an honour actually, to be in the index. You have to earn that, right. And when your pages appear in the valid, congratulations, Google thinks that your pages are worthy. You’ve done a good job, and you can relax.

• Indexed, not submitted in sitemap: 

It’s not that critical. It means that Google has found some pages Or followed links from other pages to that page. And they have included them in the index. They make that you wanted them in the index. So they put them in the index. So casually look at those pages to make sure that you want the index.

Excluded

Here is the meat and potatoes of everything in the coverage report. This is the killer. This is where I spend most of my day with our clients. This exclusive report shows every page that is failing to be indexed. And I might also say that Google finds pages on your server, every page, every resource, Javascript, Search results pages. Things that you can never find.

In this example, there are 45.100 excluded. Probably 500 are actually web pages. That’s because their Google bot finds everything on your folder, on your server. Still, you need to investigate every single category to make sure that pages that you do want to be indexed are being indexed, another good reason to keep an eye on the excluded. It’s because of bloat. If Google bot keeps coming back and it’s got to go through 45,000 pages to find the 200 that you want to index, it’s not going to go regularly. It’ll go once every three, four months instead of a normal process because it’s tired. Listen, you’ve got so much junk on your server, and we just cant Keep spending our resources going through them. So call your server, clean it up, and we can do a better job of helping you get your pages in the index.

There are quite a few categories, and it does kind of lean kind of technical. I know when I first started, it took me a couple of months to really get my hand around what each of these categories means.

The more you work with it, the more you’ll understand. And you may never see these. These are reasons why your pages can be indexed index, but you may never see any or all of these. 

What Does Each of the Excluded Mean? 

1. Excluded by No-Index Tag

That’s where most of the URLs are going. A bIt is soft, that means because there was a page error. It was a no for some reason index, but you submitted it in the sitemap.

And you should go look at those pages to see if, indeed, you didn’t want them to be indexed. Make up your mind and figure it out. And your web developer can, for those who aren’t web developers, your web developer is your friend here. Invite them. In fact, you can share the Google search console. There’s a button in the upper left right-hand corner to share what you see in the Search Console. Share that with your developer. Give them a bought lunch one day and help them run through these issues because they are technical.

2. Blocked by Page Removal Tool

This is very rare.

3. Blocked by Robots.txt 

Some of these kinds of overlap. That means that you submitted it page. But on that page, the robots.txt says, don’t index this page, so you’ve got to go and Figure it out if you will, one at that page index.

4. The 401

I’ve never seen one. And I’ve been doing this for five years, and hundreds of sites have never seen a 401.

5. Crawl Anonymously

Crawl anonymously could be anything. Google doesn’t understand to fit in any other category. It throws in that crawl anomaly, and you just have to click and look and see if there’s a pattern. Search Console is all about finding patterns because you’re not won’t necessarily want to look at each page. Like when there are 4000, you look for patterns. Maybe there’s a template file that you can fix-up. So if your site’s pretty large, just look to see if all of the pages in the blog posts are not being indexed. You’ve said don’t index posts. It can be simple.

6. The Crawled Currently Not Index

Google is saying, Hey, we went to the page. We know it’s there. We don’t care about it yet. It’s not essential. So we may be back. If you go and fix up the page, make it more important, make it more interesting for your visitors. We may indeed index it.

7. Discovered Currently Not an Index

It’s the same thing. They knew about it, but the site was overloaded, and they didn’t have a chance to really investigate it. So you’re not going to get that page in the index.

8. Alternate Page with Proper Canonical Tag

The canonicals! They’re interesting, but you only really want to care about those If you’re doing a big content audit, and these things are critical when you’re doing content audits. Because canonicals waste robots’ time. If you have two pages and the robots got to figure out which one’s more important, you make that decision. You tell Google, this is the more important page of the two that may be similar.

9. Duplicate, Google Chooses a Different Canonical Than the User

The duplicate when Google chooses a different canonical, that’s when Google has decided. But don’t make Google make your choice. You choose, and you get your canonicals in the order. And again, this is technical, and your web developer is your friend in this case.

10. Not found 404

These are not necessarily bad. It’s a page that should be 404. Let it go. But you want to briefly look in this column to make sure they should be 404. If there’s an error on the page, or if you need to redirect it, you need to pay attention because some pages get lost, the URL changes, you get redirects, and the 404 can hide a lot of interesting pages that you do want to index. So that’s another very important category along with the soft 404, hard 404. Definitely take a look at those and make sure you really want them to be in the 404 categories.

11. Page Removed Because of Legal Complaint

I have never seen that in five years. So don’t worry too much about that.

12. Page with Redirect

If a page has been told to redirect to another page, Google will not put it in the index. So, Google won’t put it in the index. If you don’t think it’s in the index should be in the index.

13. Duplicate Submitted URL is Not Selected a Canonical

This could be the case if you’ve got a lot of duplicate content and content that Google doesn’t really know what you want to do with it. So it just throws it in this category.

14. Soft 404

You just really need to look and dig deep under the cover to see why these pages are 404, and if the software for us, if it’s an empty page, if it’s a product that’s no longer in stock, make it a true 404. And don’t let it resolve to a 200 page good. 

Google Search Console Coverage Report

Back in November 2019, There was a webmasters conference out in California. It was the first one in a while. So I was in attendance, and there were a lot of other, really famous SEOs were in attendance. And we all swooped down on the various SEO representatives from Google to say, listen, we love Search Console, but it only gives us 1000 URLs for each category. We need more. Our sites have hundreds of thousands of pages, millions of pages. We need more from the Search Console than just being stuck at the 1000 page limit. 

Glen Gabe took it upon himself to be the ambassador, to keep pestering Google, to give us more. And he even did a Twitter survey to ask SEOs. What would you like to see from the Search Console?

And 58% say we’ve got to have an API to automate the things that are in the Search Console. It’s getting time-consuming. We have to click on every tab, and we can only download a thousand URLs. So let’s come on, Google. You can do better. This was in February. I think February of last year. I think he just kept pestering Google.

And the heavens opened up. Hallelujah! This was a great day around the world for technical SEOs because Google, on January 31 of this year, listened to all of the SEOs, and released the URL Inspection Tool Application Protocol Interface. And that means that we can finally automate what we see in Search Console and throw it into our favorite tool. Download everything that Google sees about our sites in specific categories, take it, mold it and make it easy to read. Put it in fancy graphs. 

This is like Christmas all over again for the SEO world. And everybody jumped on board immediately. I think one of the first solutions was from Screaming Frog. Screaming Frog crawls the site and tells you everything that’s wrong with it.

So Screaming Frog jumped on it, grabbed that API, and now, you can connect Search Console with the Screaming Frog and bring down everything that was previously only in the coverage report. It’s just a beautiful thing, and you can take it, pull it, bring it into excel and have at it. Screaming Frog gives you the data, and it allows you to pull it down in Excel. But of course, also there’s a similar, really to corporate service Sitebulb. I think a lot of the other large software. SEO vendors are going to be jumping on board, but Screaming Frog and Sitebulb were the first. Screaming Frog gave you the data, and it allows you to pull it down in excel, whereas Sitebulb is Sitebulb. They found a way to make this stuff pretty. This is something that’s shareable with your clients, and it’s easy for you to understand.

Also, various assorted SEOs took it upon themselves to just get all jolly about this and develop tools right away. This is one of the first ones that jump out. I think the very next day, the very next morning after the announcement. Valentyn Platzer developed one where online, you take your URLs, and you throw them into his box and connect with a Search Console, and all of the errors come out in a beautiful chart that you can just download.

MyDomain.Dev, They developed a beautiful chart. You just do the same thing. Basically, you connect with Google Search Console with that API, and you can see all of the problems with your site. See when those pages are indexed and the state of everything on your website. A lot of people develop Google sheets, and I love Google sheets. Cause most times, they’re just free, and people just share. SEO is a very sharing world. 

And also, there is Mike Richardson’s sheet. It’s another one. You just connect the API interface and then walk you through it. Don’t get freaked out by learning how to connect in API. Every one of these tools and spreadsheets, and services take you step by step through how to connect your API in the Search Console with their specific tool. And I use this occasionally because it’s free.

And for those watching who are techie or who do have developer friends who are going to help them, lots of people have written scripts for nodes for JavaScript and Python, and two of the more notable ones are the ones from Jose and JC Chouinard. These are two of the top guys who are doing stuff in SEO. That’s very valuable, and we thank them so much. You can choose which one you like and take that data and mold it and do it anyway, whichever way you’d like.

And the queen of the data studio. I think she was on the show a couple of weeks ago, Lazarina Stoy. She is brilliant. She has connected the Google Search Console API with Google Data Studio, and there’s nothing prettier. Then what that data does and how it is presented, and how it is clickable alterable, and shareable with your clients. And it makes you want to go here every day. Honestly, it’s just so gorgeous to see what your site is doing daily. And this is free. There is no money involved. 

How Often Do You Need to Go in and Check Your Covers Report?

Well, remember that Google doesn’t show a hundred percent accurate information. Google is Google. They kind of keep some things to themselves. So we think we don’t know for sure, but most SEOs think that Search Console data is 15, 20, sometimes 30 days behind what is actually happening on your website. So take that into consideration. If you post something and then immediately the next day, you go to see indexed in the Search Console. It is not going to be like that. So chill out and just give it patients. What you want to look for are patterns to make sure that the number of pages that are being indexed is gradually increasing and definitely not decreasing.

So you check as often as you need to establish that the pattern is going up, go to that green validation tab and ensure that that number stays the same. Suppose you’re not doing any new content or increasing. Maybe initially, when you launch a site, I look at it every free week or so. But after you’ve got a pattern established maybe once a month, every three, four weeks. Google is very nice here. If something dramatic is happening on your website, they will send you an email. In fact, they were sending so many emails that I turned them off because, for every little issue that cropped up, the owner of the search console Installation got an email. 

If you’re really not tracing or tracking your indexing, you can kind of check the console once a month, once every couple of months, because Google is going to let you know something really bad is happening on your site. But that being said. Now that we have the application protocol interface, we can move that data into the tools we use every day.

You can look at your Search Console results every single day as a matter of rote activities that you do every day. So that application API is opening up. It was just a gift from Google.

Here is the our latest recap of our latest Weekly SEO about Google Data Studio!

Also, You can watch our next Weekly SEO Stream!