How to Make Your Website AI-Friendly (What Actually Works in 2026)

“AI SEO” has become a product category, and most of what is sold under that label cannot be measured. I spent August rebuilding this site for AI discoverability, and the useful part of that work was not the tactics — it was sorting the recommendations that are actually enforced by AI crawlers from the ones that just sound plausible.

Here is what survived that sort, in the order that matters.

First, understand the three ways AI reaches your content

#

Every major AI operator runs several crawlers with different jobs, and they are governed independently. This is the detail that trips up most site owners: blocking or allowing one does nothing to the others.

The distinction is a business decision, not a technical one. Training gets you into the model’s weights with no attribution and no referral traffic. Search and user-initiated fetches are what produce a cited link back to you.

If you want to be recommended when someone asks a general question, allow all three. If your content is your product, block training and allow the other two.

Step 1: Confirm you are not already blocking them

#

This is the step that gates everything else, and it is the one people skip because it lives in a dashboard rather than in the codebase.

A block at your CDN or WAF happens at the network level. The crawler gets a 403 and never reaches the robots.txt you carefully wrote. All the structured data in the world is worthless behind that.

If you are on Cloudflare, this is time-sensitive. On September 15, 2026, Cloudflare applies new default AI bot policies: bots classified as Training or Agent get blocked on pages displaying ads, and mixed-purpose crawlers that combine Search and Training are blocked by any configuration that blocks training. Free-plan customers who have never touched these settings are moved onto the new defaults automatically.

Two settings deserve specific attention:

• Managed robots.txt — when enabled, Cloudflare serves its own robots.txt containing Disallow rules for known AI crawlers plus a Content Signals policy defaulting to ai-train=no. That silently overrides whatever your site generates.
• The legacy “Block AI bots” toggle — deprecating on the same date, and it now also blocks mixed-purpose crawlers. That includes Googlebot.

Test it from outside, with a real bot user agent. All of these should return 200:

curl -sI -A "GPTBot/1.0"        https://example.com/ | head -1
curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -1
curl -sI -A "ClaudeBot/1.0"     https://example.com/ | head -1

A 403 means your edge configuration is the problem, and no amount of on-site work will fix it.

Step 2: Name each bot in robots.txt

#

A bare User-agent: * / Allow: / is technically sufficient. Naming each crawler is still worth doing, because it makes your policy explicit, self-documenting, and robust against wildcard-matching quirks in individual crawlers.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Content-Signal: search=yes, ai-input=yes, ai-train=yes

Sitemap: https://example.com/sitemap.xml

That Content-Signal line is Cloudflare’s Content Signals Policy, which expresses intent separately for search indexing, real-time answer generation (ai-input), and model training.

Unlike much of this space, robots.txt compliance is genuinely documented by every major operator — OpenAI, Anthropic, Google, and Perplexity all publish that they honor it.

One warning if you use a static site generator: many themes deliberately emit Disallow: / in development builds. Check the file your production build produces, not the one your local server serves.

Step 3: Structured data — the real visibility lever

#

This is where most sites leave the largest gain on the table.

Nearly every CMS and theme emits something like this:

"author": { "@type": "Person", "name": "Michael Michalak" }

An AI system cannot do anything useful with that. It is a name string, not an entity. There is nothing to resolve it against, nothing to verify, and no relationship between the author of an article and the person the site is about.

What you want instead is a single JSON-LD @graph with cross-referenced @id values, so every node on every page points at the same canonical entity:

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Person",
      "@id": "https://example.com/#person",
      "name": "Michael Michalak",
      "jobTitle": "Backend Developer, Fractional CTO, AI Consultant",
      "sameAs": [
        "https://www.linkedin.com/in/michaelmichalak",
        "https://github.com/mikemichalak"
      ],
      "knowsAbout": ["Drupal", "Drupal Commerce", "AI Agents"],
      "alumniOf": { "@type": "CollegeOrUniversity", "name": "Purdue University" }
    },
    {
      "@type": "WebSite",
      "@id": "https://example.com/#website",
      "publisher": { "@id": "https://example.com/#person" }
    },
    {
      "@type": "BlogPosting",
      "author": { "@id": "https://example.com/#person" },
      "publisher": { "@id": "https://example.com/#person" }
    }
  ]
}

The @id values are the whole point. They turn a set of unrelated per-page snippets into one graph that consistently says this article was written by this specific person, who is the subject of this site, and who is also these two external profiles.

Two things to get right:

• Use the type that matches the page. A page about a person should be ProfilePage with mainEntity pointing at your Person. Pages describing what you offer should be Service, not a generic Article. Add BreadcrumbList so the hierarchy is explicit.
• Make sameAs reciprocal. sameAs is only a claim until the external profile links back. Put your site URL in your LinkedIn and GitHub profile fields. Unverifiable claims in structured data are worse than absent ones.

Keep the facts in one data file rather than scattered through templates, and only assert things a human can confirm from visible page content.

Validate with the Google Rich Results Test and the Schema.org validator after deploying.

Step 4: Make your freshness signals true

#

Check your own site for this one, because it is almost universal and completely invisible.

Across the 76 pages here, zero set a modification date. Hugo silently fell back to the publish date, which meant every dateModified in the structured data and every <lastmod> in the sitemap was just restating when the page was first written. A page I had substantially revised looked untouched since 2019.

In Hugo the fix is enableGitInfo = true, which derives modification dates from the last commit that touched each file. Every generator has an equivalent.

The catch worth knowing: this needs full git history at build time. Most hosts, including Cloudflare Pages, clone with depth 1 by default. If all your lastmod values collapse to the build date, that is why.

While you are in there:

• Create real section landing pages. /posts/ and /experience/ were rendering untitled and undescribed here, and both are prime targets for category-level queries.
• Submit your sitemap to Google Search Console and Bing Webmaster Tools. AI answer engines lean on conventional search indexes far more than on anything AI-specific, and Bing’s index still feeds several AI answer products.

Step 5: Shape content so a passage can stand alone

#

An answer engine does not cite pages. It extracts passages. Content that requires reading three sections to assemble one answer does not get used.

• Fix your heading hierarchy. If your generator renders the front matter title as the H1, a # heading in the body creates a second one and flattens the structure parsers use to segment a page. Seven pages here had this, one of them using # for every single section.
• Add question-and-answer pairs. Q&A is the single most extractable shape there is. Pair the visible content with FAQPage JSON-LD generated from the same source, so the markup can never describe a question a visitor cannot see — which is what Google requires.
• Lead with the answer. Put a complete, standalone summary in the opening block so an extractive summarizer finds a finished answer instead of having to synthesize one.
• Write descriptions that add information. A description that restates the title gives an AI system nothing it did not already have from the heading. Write it as a self-contained answer to the question the page addresses.

Step 6: Machine-readable endpoints, with realistic expectations

#

Two things here, and they are not equally worthwhile.

Markdown twins of every page: worth it. Hugo 0.164 ships a built-in markdown output format, so every page gets a clean .md version at /posts/slug/index.md, discoverable through the <link rel="alternate" type="text/markdown"> tag. An agent gets source Markdown with its lists, code blocks, and heading structure intact, rather than HTML that has been crudely stripped of tags.

llms.txt: near-zero value, near-zero cost. I want to be direct about this because the file gets recommended constantly:

• An Ahrefs study of 137,000 domains found 97% of llms.txt files received zero requests in May 2026.
• Google stated in June 2026 that the file is not required for Search.
• OpenAI’s and Anthropic’s crawler documentation both point site owners to robots.txt instead.

Ship it anyway if it costs you one build step, because AI coding agents like Cursor, Claude Code, and Copilot do read it. Just do not expect citations from it, and do not let anyone sell you an AI SEO package whose centerpiece it is.

A note on auth.md, which is not an SEO file

#

One more file has entered this conversation recently, and it belongs in a different category from everything above.

auth.md is an open protocol WorkOS published in May 2026. It answers a question none of the other files touch: how does an AI agent sign up for your service on behalf of a user, without a human filling in a form? The Markdown file at https://yourapp.com/auth.md is the prose entry point. The actual discovery path is two hops of existing OAuth standards:

1. Your API returns 401 with a WWW-Authenticate: Bearer resource_metadata="…" header.
2. That points at Protected Resource Metadata (RFC 9728) at /.well-known/oauth-protected-resource, naming the resource, its scopes, and its authorization server.
3. The authorization server’s metadata (RFC 8414) at /.well-known/oauth-authorization-server carries an agent_auth block with the registration endpoints and the flows you accept.

There are two flows. In agent verified, the agent’s identity provider signs an ID-JAG assertion vouching for the user, and the service issues a credential synchronously with no human in the loop. In user claimed, the agent shows the user a one-time code to confirm, which needs no provider integration at all.

Whether this applies to you is a yes-or-no question, not a spectrum. If your site issues credentials — accounts, an API, anything sitting behind a 401 — it is worth reading properly. If you publish content, it is not for you, and publishing the discovery documents anyway means advertising endpoints that 404. That is the same failure as an unverifiable sameAs: a machine-readable claim that does not survive being checked.

This site is squarely in the second group, so I published an /auth.md that says exactly that — no registration exists, no credential is needed, here are the open endpoints, do not go looking for a register_uri. An agent gets its answer in one fetch instead of walking a discovery chain to a dead end.

One safety note for anyone building scanners: do not probe POST /agent/auth to test whether a site supports this. That is a registration endpoint. Hitting it can create accounts, send email, and issue live credentials. The public discovery documents are the safe source of truth, and they are the only thing worth reading unattended.

How to tell whether it worked

#

There is no rank tracker for AI answers. Everything above is input; there are only two honest feedback loops.

1. Your CDN’s AI bot analytics. Cloudflare’s AI Crawl Control reports actual AI bot request volume by operator. What you want to see is GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot appearing with non-zero counts and no blocks. Give it two to four weeks.
2. Ask the models directly, monthly. Put the question your customer would actually type into ChatGPT, Claude, and Perplexity, and note whether you get cited. That is the outcome all of this aims at, and no dashboard reports it.

The short version

#

The uncomfortable truth about AI SEO is that most of it is just technical SEO done properly, plus one genuinely new requirement: explicitly permitting AI crawlers, and describing yourself in a way a machine can resolve and verify.

If you only do three things: unblock the crawlers at your edge, build a real Schema.org entity graph with @id references and reciprocal sameAs links, and make your modification dates tell the truth. That is the 80%.

Everything else on the list is refinement, and anything not on the list is probably someone’s product.

Want a second opinion on your own setup? I do this work on client sites — crawler policy, structured data, and the technical SEO underneath it. Get in touch and I will tell you which of the six steps above you are actually missing.