Controlling Indexing with Robots Meta in SPAs

Keeping a page out of the index sounds trivial — add noindex — but in a client-rendered app the timing of when that directive appears decides whether a crawler ever sees it. This guide shows how to serve a directive a crawler reads reliably, and sits within sitemaps and indexing control for SPAs.

Problem Statement

The robots meta directive tells a crawler whether it may index a page and follow its links. In a SPA it is tempting to set that directive the same way you set the title — from a router guard or an effect that runs after the component mounts. That works for a browser, but a crawler renders on its own schedule, and if the initial HTML already contained indexable content, the crawler can snapshot the indexable version before your script injects noindex. The directive arrives, but too late to matter. This is the render-then-noindex trap: the page renders as indexable, and only afterward asks not to be indexed.

The trap is a timing problem, so the fix is a timing fix — put the directive in the response the crawler renders, not in code that runs after the render snapshot.

The render-then-noindex trap versus a directive sent up front Two timelines: in the trap, the crawler snapshots an indexable page before client JavaScript adds noindex; when the directive ships in the initial response, the crawler reads noindex before it renders. The trap — noindex added late 1. Ship indexable HTML content 2. Crawler snaps indexable state 3. JS adds noindex — too late 4. Indexed anyway ✗ Fixed — directive in the initial response 1. Response carries X-Robots-Tag / meta 2. Crawler reads it before rendering 3. Excluded correctly directive respected ✓
A directive added after render can miss the snapshot; one carried in the initial response is read before the crawler renders.

Step-by-step fix

  1. Decide the directive per route, on the server or edge. Map each route pattern to its indexing intent in one place, so the value is computed before any HTML is generated. Staging, account, search-result, and filtered routes typically get noindex.

    // robots-policy.ts
export type RobotsDirective = 'index, follow' | 'noindex, follow' | 'noindex, nofollow';

const NOINDEX_PATTERNS: RegExp[] = [
  /^\/account(\/|$)/,
  /^\/search(\/|$)/,      // internal search results
  /^\/preview\//,
  /\?.*\bsessionid=/,      // session-scoped variants
];

export function directiveFor(pathname: string, search: string): RobotsDirective {
  const url = pathname + search;
  return NOINDEX_PATTERNS.some((re) => re.test(url)) ? 'noindex, follow' : 'index, follow';
}

  2. Prefer the X-Robots-Tag header when you control responses. Because it is an HTTP header, it applies before the crawler parses a single byte of body, so render timing is irrelevant. This is the most robust option at an edge worker or origin server.

    // edge-middleware.ts — runs at the CDN edge before the app shell is served
import { directiveFor } from './robots-policy';

export function onRequest(request: Request): Response | void {
  const { pathname, search } = new URL(request.url);
  const directive = directiveFor(pathname, search);
  if (directive.startsWith('noindex')) {
    // Attach the header to the shell response for this route
    return new Response(appShellHtml, {
      headers: {
        'Content-Type': 'text/html; charset=utf-8',
        'X-Robots-Tag': directive,
      },
    });
  }
  // otherwise fall through to the normal indexable response
}

  3. Otherwise, render the meta tag into the initial HTML. On a prerendered or static host where you can shape the HTML per route but not the headers, write the directive into the head of the served document so it is present before render.

    <!-- Prerendered head for /account/settings — present in the initial response -->
<meta name="robots" content="noindex, follow">

  4. Keep the client from downgrading the directive. If the app also manages the head at runtime, make sure it does not overwrite a server-set noindex with a default index on hydration. Read the existing value and preserve a stricter one.

    // preserve-robots.ts — run during head setup, do not clobber a server noindex
function setRobots(intended: string) {
  const el = document.querySelector('meta[name="robots"]');
  const current = el?.getAttribute('content') ?? '';
  // Never relax a server-provided noindex during hydration
  if (current.includes('noindex') && !intended.includes('noindex')) return;
  if (!el) {
    const meta = document.createElement('meta');
    meta.name = 'robots';
    meta.content = intended;
    document.head.appendChild(meta);
  } else {
    el.setAttribute('content', intended);
  }
}

  5. Do not disallow the route in robots.txt. A crawler that is blocked from fetching the URL can never read its noindex, so blocking and excluding are mutually exclusive. To remove a page from the index, keep it crawlable and let the directive do the work.

Choosing how to deliver a noindex directive If you can set response headers use the X-Robots-Tag header, otherwise write the robots meta tag into the initial HTML; disallowing in robots.txt never removes a page from the index. Deliver the directive where the crawler reads it first Route needs noindex Can you set headers? yes no X-Robots-Tag header read before render — best ✓ robots meta in initial HTML present before render ✓ Not an option: robots.txt Disallow blocks the fetch, so noindex is never read ✗
Pick the header when you control responses and the meta tag when you only control HTML — but never reach for robots.txt to deindex.

Validation

Verify the directive reaches the crawler in the initial response, not just the hydrated DOM:

Four checks that the crawler read the noindex before it rendered A raw curl of the header and body confirms the directive is present before any JavaScript, URL Inspection reports indexing disallowed, a CI assertion guards the policy, and the Page indexing report confirms the outcome after deploy. Confirm the directive is in the initial response, not just the DOM 1 curl -sI and curl -s the URL X-Robots-Tag header and meta tag present before any JavaScript runs 2 Search Console URL Inspection Indexing allowed reports disallowed by a noindex directive 3 Assert the policy in CI a routing change cannot silently make a private route indexable 4 Watch the Page indexing report excluded pages move to Excluded by noindex tag after deploy direct → authoritative
The fastest check is a raw curl that must already carry the directive; the report at the bottom is authoritative but only confirms days later.

  • Check the header and raw HTML directly. curl -sI https://example.com/account/settings should show the X-Robots-Tag, and curl -s of the same URL should show the meta tag in the initial body — before any JavaScript runs.

  • Use URL Inspection in Search Console. The Indexing allowed? line and the rendered HTML confirm what the crawler actually saw; a page you excluded should report indexing as disallowed by a noindex directive.

  • Assert the policy in CI so a routing change cannot silently make a private route indexable.

    // robots-policy.test.ts
import { directiveFor } from './robots-policy';

test('private routes resolve to noindex, public ones to index', () => {
  console.assert(directiveFor('/account/settings', '') === 'noindex, follow', 'account must be noindex');
  console.assert(directiveFor('/blog/hello', '') === 'index, follow', 'blog must be indexable');
});

  • Watch the Page indexing report for pages moving to Excluded by noindex tag after deploy, confirming the directive took effect.

Common Pitfalls

  • Setting noindex in a useEffect or route guard only. The directive runs after render and can miss the snapshot — the core render-then-noindex trap. Serve it in the initial response.
  • Blocking in robots.txt and expecting deindexing. The crawler never fetches the page, so it never sees the noindex, and the URL can still appear in results from external links.
  • Client head manager resetting robots on hydration. A default index written during hydration can undo a server noindex; preserve the stricter value, a pattern related to avoiding metadata hydration pitfalls.
  • Leaving noindex URLs in the sitemap. A page you exclude should not be advertised for discovery; keep the two in sync with dynamic XML sitemap generation.
  • Conflicting directives across header and meta. If the header says index and the meta says noindex, the most restrictive wins, but the ambiguity is a maintenance hazard — pick one source per route.
Why a robots.txt Disallow does not deindex a URL A robots.txt Disallow stops the crawler from fetching the page, so the noindex directive is never read, and external backlinks can still get the URL indexed. Blocking the fetch hides the noindex the page carries robots.txt Disallow: /account Crawler skips never fetches the page noindex never read directive invisible ✗ External backlinks point at the URL URL still indexed ✗ from link signals alone
Disallowing the route means the crawler never reads the noindex, so backlink signals alone can keep the URL in the index.

Frequently Asked Questions

Why does Google index a SPA page that has a noindex tag?

Because the noindex was added by JavaScript after the page rendered its indexable content. A crawler can capture the indexable state before the client script runs, and if the page also shipped indexable HTML first, the late directive is ignored. Send the directive in the initial response instead.

Should I use the robots meta tag or the X-Robots-Tag header?

Use the X-Robots-Tag HTTP header when you can set per-route response headers at the origin or edge, because it applies before any HTML is parsed and cannot be affected by render timing. Use a server-set robots meta tag when you only control the HTML, such as a prerendered or static host.

Can I block a page from indexing with robots.txt instead?

No. Disallowing a URL in robots.txt stops the crawler from fetching it, which means it never sees a noindex directive on that page and may still index the URL from external links. To remove a page from the index, allow crawling and serve a noindex directive the crawler can read.

← Back to Sitemaps & Indexing Control for SPAs