Tech Baze Logo

Welcome, Guest: Join Techbaze Forum / / Disclaimer / Privacy / Recent / Contact Us / Publish a new post
Stats: 24 members, 813 topics. Cached on: Wednesday, 21st April 2021 at

 

10 Ways to Get Google to Index Your Site (That Actually Work)

Techbaze Forum / Computers Programming Timely Updates Trending Webmaster's / 10 Ways to Get Google to Index Your Site (That Actually Work) (100 Views)

(0) (Reply)

10 Ways to Get Google to Index Your Site (That Actually Work) by : 8:18 pm On December 6, 2020
Advertisements

If Google doesn’t index your website, then you’re pretty much invisible. You won’t show up for any search queries, and you won’t get any organic traffic whatsoever.

Advertisements

Given that you’re here, I’m guessing this isn’t news to you. So let’s get straight down to business.

Advertisements

This article teaches you how to fix any of these three problems:

Advertisements
  1. Your entire website isn’t indexed.
  2. Some of your pages are indexed, but others aren’t.
  3. Your newly-published web pages aren’t getting indexed fast enough.

But first, let’s make sure we’re on the same page and fully-understand this indexing malarkey.

Advertisements

What is crawling and indexing?

Google discovers new web pages by crawling the web, and then they add those pages to their index. They do this using a web spider called Googlebot.

Advertisements

Confused? Let’s define a few key terms.

Advertisements
  • Crawling: The process of following hyperlinks on the web to discover new content.
  • Indexing: The process of storing every web page in a vast database.
  • Web spider: A piece of software designed to carry out the crawling process at scale.
  • Googlebot: Google’s web spider.

Here’s a video from Google

Advertisements
 that explains the process in more detail:

Advertisements

When you Google something, you’re asking Google to return all relevant pages from their index. Because there are often millions of pages that fit the bill, Google’s ranking algorithm does its best to sort the pages so that you see the best and most relevant results first.

Advertisements

The critical point I’m making here is that indexing and ranking are two different things.

Advertisements

Indexing is showing up for the race; ranking is winning.

Advertisements

You can’t win without showing up for the race in the first place.

Advertisements

How to check if you’re indexed in Google

Go to Google, then search for site:yourwebsite.com

Advertisements
site search indexing

This number shows roughly how many of your pages Google has indexed.

Advertisements

If you want to check the index status of a specific URL, use the same site:yourwebsite.com/web-page-slug operator.

Advertisements
google site search web page

No results will show up if the page isn’t indexed.

Advertisements

Now, it’s worth noting that if you’re a Google Search Console user, you can use the Coverage report to get a more accurate insight into the index status of your website. Just go to:

Advertisements

Google Search Console > Index > Coverage

Advertisements
google search console valid pages

Look at the number of valid pages (with and without warnings).

Advertisements

If these two numbers total anything but zero, then Google has at least some of the pages on your website indexed. If not, then you have a severe problem because none of your web pages are indexed.SIDENOTE. Not a Google Search Console user? Sign up. It’s free. Everyone who runs a website and cares about getting traffic from Google should use Google Search Console. It’s that important. 

Advertisements

You can also use Search Console to check whether a specific page is indexed. To do that, paste the URL into the URL Inspection tool.

Advertisements

If that page is indexed, it’ll say “URL is on Google.”

Advertisements
url is on google search console

If the page isn’t indexed, you’ll see the words “URL is not on Google.”

Advertisements
page is not on google search console

How to get indexed by Google

Found that your website or web page isn’t indexed in Google? Try this:

Advertisements
  1. Go to Google Search Console
  2. Navigate to the URL inspection tool
  3. Paste the URL you’d like Google to index into the search bar.
  4. Wait for Google to check the URL
  5. Click the “Request indexing” button

This process is good practice when you publish a new post or page. You’re effectively telling Google that you’ve added something new to your site and that they should take a look at it.

Advertisements

However, requesting indexing is unlikely to solve underlying problems preventing Google from indexing old pages. If that’s the case, follow the checklist below to diagnose and fix the problem.

Advertisements

1) Remove crawl blocks in your robots.txt file

Is Google not indexing your entire website? It could be due to a crawl block in something called a robots.txt file.

Advertisements

To check for this issue, go to yourdomain.com/robots.txt.

Advertisements

Look for either of these two snippets of code:

Advertisements
1User-agent: Googlebot
2Disallow: /
1User-agent: *
2Disallow: /

Both of these tell Googlebot that they’re not allowed to crawl any pages on your site. To fix the issue, remove them. It’s that simple.

Advertisements

A crawl block in robots.txt could also be the culprit if Google isn’t indexing a single web page. To check if this is the case, paste the URL into the URL inspection tool in Google Search Console. Click on the Coverage block to reveal more details, then look for the “Crawl allowed? No: blocked by robots.txt” error.

Advertisements

This indicates that the page is blocked in robots.txt.

Advertisements

If that’s the case, recheck your robots.txt file for any “disallow” rules relating to the page or related subsection.

Advertisements
robots txt

Remove where necessary.

Advertisements

2) Remove rogue noindex tags

Google won’t index pages if you tell them not to. This is useful for keeping some web pages private. There are two ways to do it:

Advertisements

Method 1: meta tag

Pages with either of these meta tags in their <head> section won’t be indexed by Google:

Advertisements
1<meta name=“robots” content=“noindex”>
1<meta name=“googlebot” content=“noindex”>

This is a meta robots tag, and it tells search engines whether they can or can’t index the page.SIDENOTE. The key part is the “noindex” value. If you see that, then the page is set to noindex.

Advertisements

To find all pages with a noindex meta tag on your site, run a crawl with Ahrefs’ Site Audit. Go to the Indexability report. Look for “Noindex page” warnings.

Advertisements
noindex ahrefs site audit

Click through to see all affected pages. Remove the noindex meta tag from any pages where it doesn’t belong.

Advertisements

Method 2: X‑Robots-Tag

Crawlers also respect the X‑Robots-Tag HTTP response header. You can implement this using a server-side scripting language like PHP, or in your .htaccess file, or by changing your server configuration.

Advertisements

The URL inspection tool in Search Console tells you whether Google is blocked from crawling a page because of this header. Just enter your URL, then look for the “Indexing allowed? No: ‘noindex’ detected in ‘X‑Robots-Tag’ http header”

Advertisements
x robots header search console

If you want to check for this issue across your site, run a crawl in Ahrefs’ Site Audit tool, then use the “Robots information in HTTP header” filter in the Page Explorer:

Advertisements
x robots tag filter site audit

Tell your developer to exclude pages you want indexing from returning this header.

Advertisements

3) Include the page in your sitemap

A sitemap tells Google which pages on your site are important, and which aren’t. It may also give some guidance on how often they should be re-crawled.

Advertisements

Google should be able to find pages on your website regardless of whether they’re in your sitemap, but it’s still good practice to include them. After all, there’s no point making Google’s life difficult.

Advertisements

To check if a page is in your sitemap, use the URL inspection tool in Search Console. If you see the “URL is not on Google” error and “Sitemap: N/A,” then it isn’t in your sitemap or indexed.

Advertisements
url not on google or sitemap

Not using Search Console? Head to your sitemap URL—usually, yourdomain.com/sitemap.xml—and search for the page.

Advertisements
sitemap search

Or, if you want to find all the crawlable and indexable pages that aren’t in your sitemap, run a crawl in Ahrefs’ Site Audit. Go to Page Explorer and apply these filters:

Advertisements
not in sitemap indexable

These pages should be in your sitemap, so add them. Once done, let Google know that you’ve updated your sitemap by pinging this URL:

Advertisements
http://www.google.com/ping?sitemap=http://yourwebsite.com/sitemap_url.xml

Replace that last part with your sitemap URL. You should then see something like this:

Advertisements
sitemap notification received

That should speed up Google’s indexing of the page.

Advertisements

4) Remove rogue canonical tags

A canonical tag tells Google which is the preferred version of a page. It looks something like this:

Advertisements

<link rel="canonical” href="/page.html/">

Advertisements

Most pages either have no canonical tag, or what’s called a self-referencing canonical tag. That tells Google the page itself is the preferred and probably the only version. In other words, you want this page to be indexed.

Advertisements

But if your page has a rogue canonical tag, then it could be telling Google about a preferred version of this page that doesn’t exist. In which case, your page won’t get indexed.

Advertisements

To check for a canonical, use Google’s URL inspection tool. You’ll see an “Alternate page with canonical tag” warning if the canonical points to another page.

Advertisements
alternate page with canonical

If this shouldn’t be there, and you want to index the page, remove the canonical tag.IMPORTANT

Advertisements

Canonical tags aren’t always bad. Most pages with these tags will have them for a reason. If you see that your page has a canonical set, then check the canonical page. If this is indeed the preferred version of the page, and there’s no need to index the page in question as well, then the canonical tag should stay.

Advertisements

If you want a quick way to find rogue canonical tags across your entire site, run a crawl in Ahrefs’ Site Audit tool. Go to the Page Explorer. Use these settings:

Advertisements
canonicals filter site audit

This looks for pages in your sitemap with non-self-referencing canonical tags. Because you almost certainly want to index the pages in your sitemap, you should investigate further if this filter returns any results.

Advertisements

It’s highly likely that these pages either have a rogue canonical or shouldn’t be in your sitemap in the first place.

Advertisements

5) Check that the page isn’t orphaned

Orphan pages are those without internal links pointing to them.

Advertisements

Because Google discovers new content by crawling the web, they’re unable to discover orphan pages through that process. Website visitors won’t be able to find them either.

Advertisements

To check for orphan pages, crawl your site with Ahrefs’ Site Audit. Next, check the Links report for “Orphan page (has no incoming internal links)” errors:

Advertisements
orphan pages

This shows all pages that are both indexable and present in your sitemap, yet have no internal links pointing to them.IMPORTANT

Advertisements

This process only works when two things are true:

Advertisements
  1. All the pages you want indexing are in your sitemaps
  2. You checked the box to use the pages in your sitemaps as starting points for the crawl when setting up the project in Ahrefs’ Site Audit.

Not confident that all the pages you want to be indexed are in your sitemap? Try this:

Advertisements
  1. Download a full list of pages on your site (via your CMS)
  2. Crawl your website (using a tool like Ahrefs’ Site Audit)
  3. Cross-reference the two lists of URLs

Any URLs not found during the crawl are orphan pages.

Advertisements

You can fix orphan pages in one of two ways:

Advertisements
  1. If the page is unimportant, delete it and remove from your sitemap.
  2. If the page is important, incorporate it into the internal link structure of your website.

6) Fix nofollow internal links

Nofollow links are links with a rel=“nofollow” tag. They prevent the transfer of PageRank to the destination URL. Google also doesn’t crawl nofollow links.

Advertisements

Essentially, using nofollow causes us to drop the target links from our overall graph of the web. However, the target pages may still appear in our index if other sites link to them without using nofollow, or if the URLs are submitted to Google in a Sitemap.

In short, you should make sure that all internal links to indexable pages are followed.

Advertisements

To do this, use Ahrefs’ Site Audit tool to crawl your site. Check the Links report for indexable pages with “Page has nofollow incoming internal links only” errors:

Advertisements
nofollow incoming links error

Remove the nofollow tag from these internal links, assuming that you want Google to index the page. If not, either delete the page or noindex it.

Advertisements

7) Add “powerful” internal links

Google discovers new content by crawling your website. If you neglect to internally link to the page in question then they may not be able to find it.

Advertisements

One easy solution to this problem is to add some internal links to the page. You can do that from any other web page that Google can crawl and index. However, if you want Google to index the page as fast as possible, it makes sense to do so from one of your more “powerful” pages.

Advertisements

Why? Because Google is likely to recrawl such pages faster than less important pages.

Advertisements

To do this, head over to Ahrefs’ Site Explorer, enter your domain, then visit the Best by links report.

Advertisements
best by links ahrefs blog 2

This shows all the pages on your website sorted by URL Rating (UR). In other words, it shows the most authoritative pages first.

Advertisements

Skim this list and look for relevant pages from which to add internal links to the page in question.

Advertisements

For example, if we were looking to add an internal link to our guest posting guide, our link building guide would probably offer a relevant place from which to do so. And that page just so happens to be the 11th most authoritative page on our blog:

Advertisements
link building guide ahrefs best by links

Google will then see and follow that link next time they recrawl the page.PRO TIP

Advertisements

Paste the page from which you added the internal link into Google’s URL inspection tool. Hit the “Request indexing” button to let Google know that something on the page has changed and that they should recrawl it as soon as possible. This may speed up the process of them discovering the internal link and consequently, the page you want indexing.

Advertisements

8) Make sure the page is valuable and unique

Google is unlikely to index low-quality pages because they hold no value for its users. Here’s what Google’s John Mueller said about indexing in 2018:https://platform.twitter.com/embed/index.html?creatorScreenName=JoshuaCHardwick&dnt=false&embedId=twitter-widget-0&frame=false&hideCard=false&hideThread=false&id=948544364090970112&lang=en&origin=https%3A%2F%2Fahrefs.com%2Fblog%2Fgoogle-index%2F&siteScreenName=ahrefs&theme=light&widgetsVersion=ed20a2b%3A1601588405575&width=550px

Advertisements

He implies that if you want Google to index your website or web page, it needs to be “awesome and inspiring.”

Advertisements

If you’ve ruled out technical issues for the lack of indexing, then a lack of value could be the culprit. For that reason, it’s worth reviewing the page with fresh eyes and asking yourself: Is this page genuinely valuable? Would a user find value in this page if they clicked on it from the search results?

Advertisements

If the answer is no to either of those questions, then you need to improve your content.

Advertisements

You can find more potentially low-quality pages that aren’t indexed using Ahrefs’ Site Audit tool and URL Profiler. To do that, go to Page Explorer in Ahrefs’ Site Audit and use these settings:

Advertisements
find low quality pages site audit

This will return “thin” pages that are indexable and currently get no organic traffic. In other words, there’s a decent chance they aren’t indexed.

Advertisements

Export the report, then paste all the URLs into URL Profiler and run a Google Indexation check.

Advertisements
url profiler

IMPORTANT

Advertisements

It’s recommended to use proxies if you’re doing this for lots of pages (i.e., over 100). Otherwise, you run the risk of your IP getting banned by Google. If you can’t do that, then another alternative is to search Google for a “free bulk Google indexation checker.” There are a few of these tools around, but most of them are limited to <25 pages at a time.

Advertisements

Check any non-indexed pages for quality issues. Improve where necessary, then request reindexing in Google Search Console.

Advertisements

You should also aim to fix issues with duplicate content. Google is unlikely to index duplicate or near-duplicate pages. Use the Duplicate content report in Site Audit to check for these issues.

Advertisements

9) Remove low-quality pages (to optimize “crawl budget”)

Having too many low-quality pages on your website serves only to waste crawl budget.

Advertisements

Here’s what Google says on the matter:

Advertisements

Wasting server resources on [low-value-add pages] will drain crawl activity from pages that do actually have value, which may cause a significant delay in discovering great content on a site.

Think of it like a teacher grading essays, one of which is yours. If they have ten essays to grade, they’re going to get to yours quite quickly. If they have a hundred, it’ll take them a bit longer. If they have thousands, their workload is too high, and they may never get around to grading your essay.

Advertisements

Google does state that “crawl budget […] is not something most publishers have to worry about,” and that “if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently.”

Advertisements

Still, removing low-quality pages from your website is never a bad thing. It can only have a positive effect on crawl budget.

Advertisements

You can use our content audit template to find potentially low-quality and irrelevant pages that can be deleted.

Advertisements

10) Build high-quality backlinks

Backlinks tell Google that a web page is important. After all, if someone is linking to it, then it must hold some value. These are pages that Google wants to index.

Advertisements

For full transparency, Google doesn’t only index web pages with backlinks. There are plenty (billions) of indexed pages with no backlinks. However, because Google sees pages with high-quality links as more important, they’re likely to crawl—and re-crawl—such pages faster than those without. That leads to faster indexing.

Advertisements

We have plenty of resources on building high-quality backlinks on the blog.

Advertisements

Indexing ≠ ranking

Having your website or web page indexed in Google doesn’t equate to rankings or traffic.

Advertisements

They’re two different things.

Advertisements

Indexing means that Google is aware of your website. It doesn’t mean they’re going to rank it for any relevant and worthwhile queries.

Advertisements

That’s where SEO comes in—the art of optimizing your web pages to rank for specific queries.

Advertisements

In short, SEO involves:

Advertisements
  • Finding what your customers are searching for;
  • Creating content around those topics;
  • Optimizing those pages for your target keywords;
  • Building backlinks;
  • Regularly republishing content to keep it “evergreen.”

Here’s a video to get you started with SEO:

Advertisements

Final thoughts

There are only two possible reasons why Google isn’t indexing your website or web page:

Advertisements
  1. Technical issues are hindering them from doing so
  2. They see your site or page as low-quality and worthless to their users.

It’s entirely possible that both of those issues exist. However, I would say that technical issues are far more common. Technical issues can also lead to the auto-generation of indexable low-quality content (e.g., problems with faceted navigation). That isn’t good.

Advertisements

Still, running through the checklist above should solve the indexation issue nine times out of ten.

Advertisements
Advertisements

Just remember that indexing ≠ ranking. SEO is still vital if you want to rank for any worthwhile search queries and attract a constant stream of organic traffic.

Advertisements
Advertisements

Reply


(0) (Reply)

How to Start a Poultry Farming with Less Income to Invest / Top 20 Richest Football Clubs In 2021; Juventus Is Ranked 10th, See Barcelona’s Rank / How to Fix NMFB CBN Covid-19 TCF Loan Application Invalid bvn Issues / Recent Liturgical Calendar 2021 / List Of Universities That Accept Jupeb In Nigeria 2021 /  

(Go Up)


Sections: Activator (23) Admission lists (6) Android Apps (20) APPS UPDATES (3) Bece (2) Books (3) Camera apps (1) Church (1) Computers (11) Film Actors (5) Free news (45) Game (109) Good Morning Sms (4) GOSPEL (1) Group links (1) How to make money (1) Illegal (1) Illegal Dating (3) Illegal Hustles (11) Illegal scammers (1) Internet Bundles (12) JAMB UPDATES (11) Jupeb (1) Latest updates (117) legal Hustles (7) Love (5) MTN UPDATES (8) Music (6) Neco (11) NECO UPDATES (2) Net Worth (167) Network (25) News (5) Phone (63) POLITICS (5) Programming (4) SCHOOLS UPDATES (20) science and technology (5) Sports (2) SPORTS UPDATES (19) Tech (26) TECHNOLOGY NEWS (4) Timely Updates (88) Trending (102) Uncategorized (41) Universities (1) Video (5) VIDEOS (2) Vpn (6) Waec (69) WAEC RUNZ (64) WAEC UPDATES (12) Webmaster's (8)

Links: (0) (1) (2) (3) (4) (5) (6) (7) (8) (9)

Techbaze Forum - Copyright © 2019 - Forumer. All rights reserved. See Own a Website Today.
Disclaimer: Every member is solely responsible for anything that he/she posts or uploads on Techbaze.