A growing share of visits are not people. They are agents acting for people — comparing, extracting, sometimes transacting. Most sites are accidentally hostile to them.
Agentic browsing is early, but the trajectory is clear enough to design for. An assistant asked to “find me a conveyancing solicitor in Leeds with fixed fees” will visit sites, extract facts and compare them. Whether yours survives that process is now a design question.
What blocks agents today
- Critical facts — price, location, availability — rendered only after client-side JavaScript, or behind an interaction.
- Key information locked in PDFs or images with no text equivalent.
- Contact and pricing details available only via a form submission.
- Aggressive bot mitigation that cannot distinguish a user’s assistant from a scraper.
- Interstitials and consent walls with no accessible dismissal path.
What makes a site legible
The good news is that the fixes overlap almost entirely with accessibility and technical SEO. Server-render the facts that matter. Mark up products, services, prices, opening hours and locations with schema.org. Use semantic HTML so structure is inferable without visual layout. Publish price ranges rather than hiding them behind “contact us”.
None of this is speculative work for a hypothetical future. Every item on that list also improves how you appear in traditional search results and answer engines today.
The llms.txt convention
A lightweight convention is emerging: an llms.txt file at your root that points AI systems to your most useful, canonical content in plain Markdown. It is not a standard yet and no engine is obliged to honour it, but it costs almost nothing to publish and makes your intent explicit.
Decide your policy deliberately
You may not want agents extracting everything, and that is a legitimate position. The point is to make it a decision rather than an accident of your bot rules. Set robots directives per user agent, allow what serves you, and block what does not — but know which of those you are doing.