2026-08-03 · Jack Stovell
Preparing your site for AI agents
An AI agent visiting your site doesn't browse — it fetches your HTML, extracts what it needs, compares it against alternatives, and moves on. Preparing for agents therefore means, almost entirely, the things that already serve crawlers well: server-rendered HTML, structured data, explicit facts and stable URLs. The genuinely new parts, like MCP endpoints for personal data, are worth understanding but are early-adopter territory as of mid-2026, not this week's job.
What changes when an agent visits instead of a person?
Everything persuasive stops working and everything extractable starts mattering more. A human visitor can be won over by design, animation and tone; an agent completing a task — "find me three freelance data engineers in Manchester and summarise their experience" — parses your page for facts, weighs them against other candidates' pages, and leaves. There is no dwell time to earn and no impression to make; there is only whether the needed fact is present, explicit and parseable. Two consequences follow. First, a fact that isn't in your HTML effectively doesn't exist: "available for contract work", your location, your specialism — if they're implied by a photo or buried in a PDF, the agent moves on to a candidate who stated them. Second, rendering matters brutally. Most AI crawlers and fetchers do not execute JavaScript, so a client-side-rendered SPA hands an agent a near-empty shell — the failure mode we dissect in how sites lie to crawlers. Agents don't squint at a broken page; they take the next result.
What should you do today?
Do the crawler fundamentals, because agents inherit them. Concretely:
- Serve real HTML from the server. Semantic structure — headings, lists, plain paragraphs — is what parsers consume; our semantic HTML guide covers the specifics.
- Add Person (or Organization) JSON-LD with a sameAs array, so an agent can resolve who the page is about without inference.
- Publish an llms.txt index. It's a proposed standard with early, unproven adoption — no major vendor has committed to using it — but it costs near zero and crawlers do request it in server logs, including ours. How to write one: what is llms.txt.
- Keep URLs stable. Agents and the systems behind them revisit; a moved page is a broken fact.
- Check your robots.txt isn't blocking the agents you want. Hosts and CDNs increasingly block AI user agents by default — our free checker tests your robots.txt stance, JavaScript-shell risk, identity markup and llms.txt in one pass.
- State facts explicitly, with dates. "Available from September 2026", "based in Leeds", "10 years in embedded systems" — the sentence an agent can lift is the sentence that wins.
This is also exactly what hilyt profiles are built to be: edge-rendered semantic HTML, Person JSON-LD, dated claims with sources, and a plain-text card at i.hilyt.it/username/llm.txt — the live example shows the format an agent actually receives.
What is MCP and does it matter for a personal site?
MCP — the Model Context Protocol — is an open standard released by Anthropic in November 2024 for connecting AI systems to tools and data sources; instead of scraping a page, an assistant can query an MCP server and get structured answers. It has real adoption for software tools and business data, and you could in principle run an MCP endpoint that serves your professional profile to any assistant that connects. For most individuals, though, the honest ranking is clear: a readable public page pays off today with every agent and crawler that exists, while a personal MCP endpoint is early-adopter territory — worth watching, reasonable to experiment with if you enjoy the plumbing, and not the thing to do before your HTML is in order.
Will AI replace LinkedIn-style search?
Nobody knows, and what follows is speculation, labelled as such. The plausible direction: as assistants get better at multi-step tasks, "find me a person who..." queries increasingly get answered by agents assembling shortlists from whatever they can fetch and verify — and walled gardens are poorly positioned for that world, because assistants generally cannot read a LinkedIn profile live. If that shift happens, the people who are findable are those whose facts live on the open web in parseable form. Against that: recruiting habits are sticky, platforms can cut their own AI deals, and timelines in this space embarrass everyone who predicts them. What seems robust either way is that a readable, corroborated open-web presence is cheap insurance — it costs little, it already improves how today's AI answers questions about you, and no plausible future punishes you for having it. What this guide won't promise: that agent traffic will arrive on any schedule, or that readiness converts to selection. Being parseable removes the reasons an agent skips you; the choice among parseable candidates belongs to retrieval, ranking and model behaviour. Start with an AI-readable profile and the fundamentals above — they are the same work either way.
Create your AI-readable profile
Related guides
- How to create an AI-readable profile of yourself
- Semantic HTML for AI readability: the markup that makes you legible
- What is llms.txt — and how to write one (with a full example)