How to Run a Technical SEO Check Before You Launch

What this check is actually for

A pre-launch technical check is not an optimisation exercise. It is a search for the small number of mistakes that make a site invisible, and they are almost all binary: either the crawler can reach the page or it cannot, either the page is allowed in the index or it is not, either your addresses agree with each other or they contradict. Nothing in this list will make a page rank. Every item on it can stop one from having the chance.

What makes these worth a dedicated pass is that they fail silently. A broken payment flow produces a support ticket within the hour. A directive that removes a site from search produces nothing at all — no error, no alert, no exception in a log — until somebody notices weeks later that traffic which should have started never did. The whole point of doing this before launch is that nobody is watching the graph yet.

Can a crawler reach the pages at all

Request your key pages the way a machine would: a plain HTTP client, no browser, no JavaScript, and read what comes back. If the body arrives empty and the content is assembled client-side, if a consent wall stands in front of the text, or if a bot-protection layer challenges anything without a familiar user agent, then you have found the problem before writing a word of content. Do this for the homepage, a product page, a documentation page and a pricing page, because they are often served by different parts of the stack.

Then check reachability by navigation rather than by URL. Crawl your own site from the homepage and list every page that turns up, then compare that list against what you believe you published. Anything present in the sitemap but absent from the crawl is an orphan, connected to the rest of the site by nothing, and the fact that it exists at a valid address is not the same as it being findable.

The two lines that can remove an entire site

Read robots.txt on the production domain, out loud if it helps, and confirm it is not the staging one. A Disallow: / that survives a deploy is the single most common way a launch goes quiet, and it costs nothing to check and everything to miss. While you are there, confirm you are not blocking the stylesheet and script directories, because a crawler that cannot render the page the way a person sees it evaluates something other than your site.

Then grep the templates for noindex. It belongs on staging, on internal search results, on empty tag pages and on filter permutations; it does not belong on the production layout, and it arrives there by way of a copied template more often than anyone expects. Note also that these two controls fight each other: a page disallowed in robots.txt is never fetched, so a noindex on it is never read, and the page can stay in the index on inbound links alone. If you want something removed, it must be crawlable and carry the directive.

One page, one address

Pick one canonical form of your URLs and make everything else redirect to it — http to https, www to bare domain or the reverse, one convention on trailing slashes, lowercase paths. Then confirm that every page carries a self-referencing canonical, and that the canonical on a page is not pointing at the homepage, at a redirect, or at a URL that carries noindex. A template bug that canonicalises the whole site to one address is a small change with a very large blast radius.

Duplication also arrives from the URLs your own site generates: tracking parameters, sort orders, filter combinations, print views, session identifiers, paginated tails. You do not have to eliminate them, but you do have to decide which address is the real one and make every signal say the same thing. The canonical is a hint rather than a directive, and it is consistency with your internal links and sitemap that makes the hint hold.

The files that have to agree

A sitemap should list only URLs you would be happy to see indexed: canonical, returning 200, not blocked, not noindexed. Generate it from the same source as your routes so a page cannot exist without an entry and an entry cannot outlive its page, and reference it from robots.txt. If your build stamps every URL with today's date on every deploy, remove the field rather than shipping a signal you know is noise.

If you ship more than one language, the annotations declaring which page serves which locale have to be reciprocal — every version referencing every other, itself included — and the codes have to be right, which is where a surprising number of sites quietly fail. A language version that is linked to but does not link back is generally ignored, and the failure produces no error anywhere. Keep the mapping in the sitemap rather than in head tags on every page, and each localised page should still canonicalise to itself, never across languages.

Status codes, and what to re-check afterwards

Confirm that a missing page returns 404 rather than a styled page with a status of 200, because the second tells a crawler that every wrong address is a real page. Confirm redirects are single hops rather than chains, and that they are permanent where you mean them to be permanent. If you are replacing an existing site, the redirect map from old addresses to new ones is the highest-value item in this whole list and the one most often finished last.

After launch, three things are worth a standing check rather than a one-off. Whether the number of pages an engine has indexed matches roughly what you published. Whether new pages are getting crawled within a sensible window. And whether anything in the deployment pipeline has reintroduced a directive from a template — the failure mode is not that these settings are wrong once, it is that they are correct at launch and quietly regress two releases later, with nothing to announce it.