How a new VALORAE page gets found by search engines, and what our own sitemaps failed

On 2 October 2026 Google's Gary Illyes put a number on something site owners usually guess at.

A new URL, he reportedly said, is typically discovered in about 20 hours. The slowest cases run from weeks to never.

That figure reached us through PPC Land's report of 3 October, which is itself based on a recap by John Campbell of the agency ROAST, written from the audience at Google's Search Central Live Deep Dive event. So it is a recap of a talk, not a Google document. We are treating it as Google's direction, not Google's policy.

The word that stuck was "never". The recap ties it, more often than not, to quality.

Six VALORAE blogs publish most days. If a meaningful share of those pages sat in "never", we would want to know whether the cause was the writing or the plumbing. The writing is a longer conversation. The plumbing we could check that afternoon.

Step 1: The checklist we borrowed

The same week, PPC Land reported a study by lintlab, a vendor that sells sitemap-checking tools. It crawled 1,000 popular origins from Google's August 2026 Chrome UX Report on 1 October.

lintlab found that 62% of the sites exposing a sitemap failed at least one check. The two most common failures, by its count, were URLs in the sitemap that redirect, at 23% of sites, and identical last-modified dates repeated across files, at 22%.

A vendor has a reason to find problems. We read it that way. But its list of checks was concrete enough to run against ourselves:

• every listed URL answers 200, without a redirect
• no http addresses, only https
• no fragments, the part after a hash
• no URLs on a different host, unless that host declares the sitemap
• last-modified dates that reflect real changes, not one date stamped on everything

Step 2: What we checked, and how

On 4 October 2026 we fetched the sitemap of each of the six sites the group publishes on. Together they list 259 URLs.

Each URL was requested once with Googlebot's user agent, with redirects switched off so a redirect would show as a failure instead of quietly resolving. The group site was checked slowly, one request at a time, because its host challenges bursts of traffic.

Nothing clever. A loop and a log file.

Why switch redirects off? Because a URL that redirects still loads fine in a browser, so nobody notices. A crawler notices. It spends a request on the old address, then another on the new one, and the sitemap has told it something untrue about where the page lives.

Step 3: What passed

All 259 URLs returned 200. None redirected.

None used http. None carried a fragment.

The VALORAE Cast sitemap lists two pages on subdomains of the Cast site, which the sitemap rules normally forbid. We checked both subdomains. Each one's robots file names the main Cast sitemap, which is the method the sitemap protocol provides for exactly this case. So that one passes, though only because someone set it up properly months ago.

On the blog pages themselves, the last-modified date is the post's update date if it has one, and its publish date if it does not. It moves when the post changes and not otherwise. That is the behaviour the study rewards.

Step 4: What did not

Two things.

Last-modified dates on fixed pages. On the NovaTechRay sitemap, 18 of the 68 URLs share a single last-modified date. They are the site's fixed pages: services, methodology, contact, the industry pages. The Arc and Media sitemaps show the same pattern on a smaller scale, with 13 URLs each sharing one date.

Some of those pages may not have changed since that date. Some probably have. We cannot prove either from the sitemap, and neither can Google, which is the point of the study's warning. A date that cannot be trusted is a date a crawler learns to ignore.

The fix is to date fixed pages from their own content, the way blog posts already are. It is logged for the sites' owners. We are not going to hand-edit dates to look varied, because that would be the same problem with better camouflage.

One site missing from IndexNow. After each publish, five of the six sites are submitted to IndexNow, the protocol Bing and several other engines use to hear about new pages quickly. Google does not take part, so for Google the sitemap remains the push signal. The group site, the one you are reading, is not on that list yet. It is logged too.

Step 5: The four steps every new page goes through

This is what happens after a post on any of the six blogs clears its checks:

1. Linked. The post appears at the top of its blog index, so a crawler visiting a page it already knows finds the new one.
2. Listed. It is added to that site's sitemap with its own date, and to the site's RSS feed.
3. Verified. The live URL is requested as Googlebot and must return 200, a canonical tag pointing at itself, and the site's name in the title. It must also appear in the sitemap and the feed. A failure here blocks the report until it is fixed.
4. Resubmitted. The sitemap is resubmitted in Search Console, and Search Console's own reading of it is recorded in that day's report.

None of this makes Google index a page. The recap gives sitemap processing a typical time of about 24 hours and a slowest case of up to 14 days, "or never (quality)". A sitemap announces a page. It cannot make a crawler value it.

Why this belongs on the ledger

The group page exists to say how the group runs, including the parts that are not finished. How a VALORAE page gets corrected covers what happens when a published claim turns out wrong. This is the same idea applied to plumbing.

The sitemap check took an afternoon. It found nothing broken and two things weak. We would rather publish that than claim every page is discovered in 20 hours, because nobody outside Google can see that number for their own site.

For a practical walk-through of how discovery feeds AI answers on top of ordinary search, NovaTechRay keeps a guide to AI crawlers and robots.txt.

The short version

• Treat Google's reported 20 hour discovery figure as a typical case from a talk recap, not a promise.
• Check every sitemap URL answers 200 without a redirect. Ours did, all 259.
• Use https only, with no fragments and no undeclared cross-host URLs.
• Date each page from its own last change. Our fixed pages fail this today.
• Link every new page from a page crawlers already visit.
• Resubmit the sitemap and record what Search Console says back.

The group's working notes live at valoraehq.com.