Get pages found in search

How search meets your site

A search engine does three jobs in a row. It crawls: a program follows links and reads your pages. It indexes: it decides what each page is about and stores it. It ranks: when somebody searches, it picks from what it stored.

A failure at one step makes the later steps irrelevant. A page that cannot be crawled is never indexed. A page that is indexed but looks identical to three others may be stored and never shown.

That is why the order below matters. Findability in Growth Score is the score for exactly this path: whether search can find and list your pages. It reads your robots.txt and sitemap, your page titles and descriptions, and how your pages are addressed. You do not need an account to do the first pass by hand.

Step 1: let crawlers in

Open yoursite.com/robots.txt in a browser. It is a short text file, and it is the single most dangerous file on a website, because one line can remove the whole site from search.

Look for this line: Disallow: / under User-agent: *. That means every crawler, every page, stay out. It is often left over from a staging site that went live.

Then check three other ways a page hides itself:

• A noindex tag in the page head, or an X-Robots-Tag header, which tells search to drop the page.
• A login or cookie wall in front of content you want found.
• Content that only appears after a script runs and a click happens, so the crawler sees an empty shell.

If a page matters, view its source (not the inspector) and confirm the words you want ranked are in the HTML. Run the free robots.txt tester to see what your file really blocks, including rules you forgot you wrote.

Step 2: an honest sitemap

A sitemap is a list of the pages you want found. It does not force indexing. It tells search where to look first and which pages you consider canonical.

A sitemap helps only while it tells the truth. The usual lies:

• Addresses on http while the site lives on https.
• Pages listed twice with and without a trailing slash.
• Carts, search results, tag archives and thank-you pages that you never wanted indexed.
• A last-modified date that is simply the build time on every page, which teaches search to ignore the field.

Keep it to pages that return a normal page, are not redirected and are not blocked. A single sitemap file holds up to 50,000 addresses; bigger sites split it into several and list them in an index file.

Then point to it: add a Sitemap line to robots.txt and submit it in Google Search Console. The sitemap checker reads yours and names each lie.

Step 3: one URL per page

Search treats every different address as a different page. When one page lives at several addresses, its strength splits between them and search has to guess which to show.

The common splits:

• www and non-www both answering.
• http and https both answering.
• Capital letters: /Pricing and /pricing.
• Tracking parameters that produce a new address for each campaign.
• A trailing slash that works both ways.

Pick one form of each. Redirect every other form to it with a permanent (301) redirect, and add a canonical tag on each page pointing at the preferred address.

While you are there, read your addresses as a visitor would.

Short, lowercase, words separated by hyphens, no dates that will make the page look stale in a year. /pricing beats /index.php?id=47 for the person who reads the link and for the engine that does.

Step 4: titles and descriptions that earn the click

Once a page is indexed, the title and description are your pitch in the results. Search often rewrites them, but it starts from yours, and a missing or duplicated one makes pages look interchangeable.

Write one of each per page, by hand.

Step 5: link your pages

Crawlers move by links. A page nothing links to (an orphan) can sit in your sitemap and still be treated as unimportant, because nothing on your own site vouches for it.

Give every page you care about at least one real link from another page, ideally from a page that already gets visits.

Use descriptive link text: "our guide to checkout leaks" tells search and a reader what is behind the link; "click here" tells nobody anything.

A simple routine: list your five most important pages. For each, find three pages where a natural sentence could point at it, and add the link.

Do not stuff links into footers and call it done. A link inside a paragraph that is about the same subject carries more meaning than a row of fifty in a menu.

Step 6: check what was stored

Everything above is theory until you look at what search reports. Connect Google Search Console (it is free and read only) and open the page indexing report. It lists pages that are indexed and, more usefully, pages that are not, each with a reason.

The reasons worth acting on:

• Noindex: confirm it was on purpose.
• Blocked by robots.txt: see Step 1.
• Duplicate without a user-selected canonical: go back to Step 3.
• Crawled, currently not indexed: search read the page and was not impressed. Usually thin content, or a page too similar to another.
• Discovered, currently not indexed: search knows the address and has not got to it yet. Better internal links help.

Use the URL inspection tool on your three most important pages and read the verdict aloud. If it says the page is not on Google, you have found your first job.

A worked example

Picture a small shop that sells coffee gear. It launched on a staging copy of the site and forgot one line. Its robots.txt still says Disallow: / from the build days. Nothing ranks. The owner rewrites product pages for months.

The path above finds it in minutes. Step 1 shows the line. After the fix, Step 2 reveals a sitemap listing cart and filter pages.

Step 3 shows /grinders and /Grinders both live. Step 4 shows every product title reads "Product | Shop". Step 5 shows the best-selling grinder is linked from nowhere.

None of that needed a new tool or a bigger budget. It needed somebody to walk the route in order. After each fix, the owner re-runs the page check and reads the indexing report a week later to see the status move.

Make it a habit, not a project

Findability breaks quietly. A redesign changes addresses. A plugin adds a noindex. A developer copies a robots file between environments. Nobody notices until traffic drops.

Three habits catch most of it:

1. 1Re-read robots.txt and your sitemap after every deploy that touches routing or templates.
2. 2Check the page indexing report monthly for new exclusions.
3. 3When you launch a page, write its title and description before it goes live, not after.

Growth Score reads these signals every day and tells you when a test flips, so a bad deploy does not sit unseen for a quarter. The free Growth Diagnosis shows where your site stands today.