Google Crawled Your Page. That Doesn't Mean It Kept It.

There's a moment in every SEO audit where someone says: "Google has crawled all our pages, so we're fine."

No. Crawling and indexing are two completely different decisions. Crawling means Google visited the page. Indexing means Google decided the page was worth remembering. If you haven't read how Google crawls a website yet, start there — crawling is the step before everything in this article.

Google crawls billions of pages. It indexes a fraction of them. The rest get visited, evaluated, and quietly discarded. Your page might be one of them — and you'd never know unless you checked.

What Indexing Actually Means

When Google indexes a page, it creates a compressed entry in its database. That entry contains:

The content — not a copy of your HTML, but Google's processed understanding of what the page says. It extracts the main text, identifies the topic, maps the entities mentioned, and stores a semantic representation of the page's meaning.

The metadata — title tag, meta description, canonical URL, language, structured data. All the signals that help Google understand how to present and categorise the page.

The relationships — which pages link to this one, which pages this one links to, which topic cluster it belongs to, how it relates to other pages Google has indexed on the same subject.

The quality signals — experience markers, authority indicators, freshness, factual accuracy, content depth relative to competing pages on the same topic. These map directly to what Google calls E-E-A-T — Experience, Expertise, Authoritativeness, and Trust. Understanding how Google ranks pages helps you see why these quality signals determine whether your indexed page appears on page 1 or page 10.

Think of the index as Google's memory. Crawling is seeing something. Indexing is deciding it's worth remembering. And just like human memory, Google is selective about what it keeps.

To understand why Google is selective, you need to look at how search engines actually evaluate information—this deep dive into how search engines think explains that decision-making layer.

Why Google Doesn't Index Everything It Crawls

Google has explicitly stated that it doesn't index every page it discovers. The reasons fall into a few categories:

The Page Told Google Not To

Noindex tag: If your page has <meta name="robots" content="noindex"> in the HTML head, Google will crawl it but won't index it. This is deliberate — you're telling Google to forget this page. Common uses: thank-you pages after form submissions, internal search results pages, staging environments.

Canonical tag pointing elsewhere: If your page has <link rel="canonical" href="https://yoursite.com/other-page">, you're telling Google: "This page is a copy. The real version is over there." Google will typically index the canonical URL and skip yours.

The canonical tag is one of the most misunderstood elements in SEO. It doesn't redirect users. It doesn't block crawling. It tells Google which version of a page is the "official" one when multiple versions exist. If you have the same product accessible at three different URLs (with different sorting parameters, for example), the canonical tag tells Google which one to index.

Common mistake I see in audits: Pages with a canonical tag pointing to a completely different, unrelated page. This usually happens when a CMS template applies a default canonical incorrectly. The result: Google ignores the page entirely because it thinks it's a copy of something it isn't.

The Page Wasn't Good Enough

Google evaluates quality before indexing. A page that's thin (barely any content), duplicate (substantially similar to another indexed page), or low-value (doesn't add anything to what's already in the index) gets crawled and then dropped.

In Google Search Console, these show up as "Crawled — currently not indexed." Google visited. Google read the content. Google decided: not worth storing.

This is the most frustrating indexing status because there's no specific error. Google just didn't think the page added enough value. The fix is almost always: make the content genuinely better, more comprehensive, more specific, or more useful than what's already indexed for the same topic.

Google Hasn't Gotten Around to It Yet

"Discovered — currently not indexed" means Google knows the URL exists (found it in your sitemap or through a link) but hasn't crawled it yet. This usually resolves on its own — Google has a queue, and your page is in it.

But if a page stays in "Discovered" status for weeks or months, it's a signal that Google doesn't consider it a priority. This often happens with pages that have few or no internal links pointing to them. Google sees the URL but has no reason to believe it's important enough to jump the queue.

Duplicate Content and the Indexing Decision

Duplicate content is the single biggest indexing problem on most websites. And it's usually not intentional.

What counts as duplicate: Two pages that are substantially similar in content. Not identical — substantially similar. If 80% of the content on Page A also appears on Page B, Google considers them duplicates and will typically only index one.

Where duplicates come from:

URL parameters create duplicates silently. yoursite.com/shoes and yoursite.com/shoes?color=red&sort=price might show the same content — or very similar content with minor filtering changes. Google sees them as separate URLs with duplicate content.

HTTP vs HTTPS, www vs non-www. If both http://yoursite.com and https://yoursite.com serve the same content (without a redirect), that's duplicate content. Same with www.yoursite.com and yoursite.com.

Pagination can create duplicates. Page 1, page 2, page 3 of a blog archive often share the same meta tags and introductory text, with only the listed articles changing.

How Google handles duplicates: It picks what it considers the "best" version and indexes that one. The others get marked as duplicates and excluded from the index. Google's choice of "best version" depends on canonical tags (if present), which version has more backlinks, which version gets more traffic, and which URL format Google prefers.

The problem: When Google picks wrong. If it indexes the parameterised URL instead of the clean URL, or the HTTP version instead of HTTPS, your SEO signals get split and your page underperforms. Proper canonical tags and redirects prevent this.

How to Check What's Indexed (And What Isn't)

Method 1: Site Search

Type site:yoursite.com into Google. This shows every page Google has indexed from your domain. Compare the number of results to the number of pages you expect to be indexed. If you have 200 pages but Google shows 50 results, 150 pages aren't indexed.

Refine with specific queries: site:yoursite.com/blog/ to check just your blog. site:yoursite.com inurl:product to check product pages.

Method 2: Google Search Console — Pages Report

Go to Search Console → Pages (or "Indexing" → "Pages"). This gives you the complete picture:

Indexed pages: Pages in Google's index. Your goal is to get all important pages here.

Not indexed — various reasons: Each excluded page has a specific reason. The most common:

• "Crawled — currently not indexed" → Google read it but didn't keep it. Quality or value problem.
• "Discovered — currently not indexed" → Google knows about it but hasn't crawled it. Priority problem.
• "Duplicate without user-selected canonical" → Google found duplicates and picked a canonical itself.
• "Duplicate, Google chose different canonical" → You set a canonical but Google disagreed.
• "Excluded by noindex tag" → Intentional (if you set it) or accidental (if a developer left it from staging).
• "Blocked by robots.txt" → Google can't crawl the page. Check if this is intentional.

Method 3: URL Inspection Tool

In Search Console, paste any URL into the search bar at the top. This tells you exactly what Google knows about that specific page: whether it's indexed, when it was last crawled, what canonical Google is using, and whether there are any issues.

This is the most precise diagnostic tool. Use it whenever a specific page isn't performing as expected.

The Noindex Trap That Destroys Rankings

Every few months I audit a site that has a mysterious ranking problem, and the cause turns out to be the same thing: noindex tags left over from a staging environment.

Here's how it happens. A developer builds the site on a staging server. They add <meta name="robots" content="noindex"> to every page so Google doesn't accidentally index the staging version. Smart move. Then they push the site to production. They forget to remove the noindex tags. Or the CMS has a "discourage search engines" setting that was checked during development and never unchecked.

The site goes live. Everything looks fine to humans. But Google reads the noindex tag and obeys it. Pages that were previously indexed start disappearing from search results. The drop is gradual — Google re-crawls pages over days and weeks, finding the noindex tag each time and removing the page from the index.

By the time someone notices the traffic drop, it's been weeks. And the fix (removing the noindex tag) takes another few weeks to fully recover because Google has to re-crawl and re-index every affected page.

How to check right now: View the source of your homepage. Search for "noindex". If you find it and didn't put it there intentionally, you have a problem that needs fixing immediately.

Structured Data and Indexing

Structured data (schema markup) doesn't directly cause a page to be indexed or not indexed. But it significantly affects how Google understands and presents your page.

A page with proper Article schema, FAQ schema, or HowTo schema gives Google explicit signals about what the content is. Google doesn't have to guess — the structured data tells it: this is an article, here's the author, here's the publication date, here are the FAQs.

Pages with rich structured data tend to get richer search results — FAQ dropdowns, how-to steps, review stars, recipe cards. These rich results get higher click-through rates, which sends positive signals back to Google, which reinforces the page's indexing and ranking.

Think of structured data as a label on a box in a warehouse. The warehouse (Google's index) has billions of boxes. A box with a clear label gets found faster, stored in the right section, and pulled out when someone asks for it. A box with no label gets shoved in a corner. Google's structured data documentation covers every schema type available — start with Article and FAQ schema for blog content.

The Practical Indexing Audit

Do this for your site right now:

Step 1: Check total indexed pages. Search site:yoursite.com in Google. Note the number. Does it match what you expect?

Step 2: Open Search Console → Pages. Read the "Not indexed" section. What are the top reasons? Are important pages being excluded?

Step 3: Check for noindex tags. View source on your 5 most important pages. Search for "noindex". If found, determine if it's intentional.

Step 4: Check canonicals. On those same 5 pages, find the canonical tag. Does it point to the correct URL? Or is it pointing somewhere wrong?

Step 5: Identify duplicates. In Search Console, look for pages marked as "Duplicate." Are they actually duplicates? Or are they unique pages that Google mistakenly considers similar?

Step 6: Find crawled-but-not-indexed pages. These are your content quality opportunities. Each one is a page Google visited and rejected. Either improve the content or remove the page entirely.

Most sites I audit have 30-50% of their pages not indexed. The site owner had no idea. They assumed that publishing a page meant Google would show it. Publishing means nothing until Google decides the page deserves a spot in its memory.

Key Takeaways

Crawling and indexing are separate decisions. Google crawls many pages it never indexes. Being crawled is not the same as being in Google's search results.

Google indexes a page only if it adds unique value to the index. Thin content, duplicate content, and low-quality pages get crawled and discarded.

Canonical tags tell Google which version of a page is the "official" one. Misconfigured canonicals are one of the most common causes of indexing problems.

Noindex tags from staging environments are a silent ranking killer. Always check for accidental noindex tags after any site migration or development push.

Use Google Search Console's Pages report and URL Inspection tool to diagnose indexing problems. Don't assume your pages are indexed — verify it.

The number of indexed pages on your site should roughly match the number of pages you actually want in search results. If there's a big gap, you have an indexing problem worth investigating.

Thinking of a career in digital marketing?
Learn from the experts at Wizgrowth who build the strategies that scale world-class brands

No fluff, just results. Chat with Academy Lead