AI & the open web

Markdown for AI Bots: Cheaper to Scrape, Unproven to Cite

APH Marketing Desk ·5 min read Share Print
In this piece
    Figure The four points the piece sets out Why this matters for publishers →
    1. 01

      The certain benefit accrues to the other side.

    2. 02

      Machine-readability is a licensing asset, not a free feature.

    3. 03

      Ungated markdown undermines your bot strategy.

    4. 04

      Nobody has measured the upside, including the pioneers.

    Marketing Desk

    The AI-optimisation tactic of the moment is disarmingly simple: publish a markdown version of your site — no JavaScript, no visuals, no ad stack, just clean structured text — so that AI systems can read you more easily. The efficiency claim is real: markdown versions cut the tokens an AI system processes by roughly 90%. And Time has gone furthest with the idea, serving ads to AI agents inside its markdown files — an experiment in monetising the machine audience directly.

    The case against arrived in the same week, and it is empirical. Promptwatch analysed 1.6 million citations and found that markdown “did not increase” the likelihood of being cited. Google’s own GEO guidance says markdown isn’t necessary for visibility in its AI surfaces. And the agent-facing ads — the genuinely novel part — have not yet shown improved advertiser results, by Time’s own account of where the experiment stands.

    Which leaves the trade in an unusual shape: concrete on one side, speculative on the other. Ship markdown and you have, with certainty, made your content dramatically cheaper for AI companies to ingest. What you get back — more citations, more AI visibility, eventually more traffic or revenue — is precisely the thing the only large-scale study on record failed to find.

    01Follow the token savings

    The operator’s question about any optimisation is: who captures the value? Here the answer is uncomfortable. A 90% reduction in tokens processed is a 90% reduction in somebody’s compute bill — and that somebody is the AI company reading you, not you. Multiplied across millions of pages and continuous recrawling, that is real money, delivered to the exact counterparties publishers are elsewhere trying to charge for access. Meanwhile the publisher’s side of the ledger — citation lift — is the speculative side. In any licensing negotiation, making yourself 90% cheaper to consume before the price discussion starts is negotiating against yourself.

    That does not make markdown worthless. It makes it an asset being given away that should be priced. If clean, structured, low-token versions of your content are valuable to AI companies — and the compute math says they are — then they belong inside the licensing conversation as a deliverable, not outside it as free SEO-adjacent plumbing. “We will serve you a machine-optimised feed” is a contract clause. Publishing it as an open endpoint is the same clause, minus the payment.

    There is also an access-control problem hiding in the tactic. This is the same week the industry spent arguing about stealth crawlers that ignore robots.txt and mimic human browsing. An ungated public markdown endpoint is a gift to exactly that population: a discount aisle for the bots you are elsewhere spending money to block. Whatever you decide about markdown, serving it indiscriminately is the worst available configuration.

    02If you test it, test it like an experiment

    The failure mode here is cultural, not technical: shipping markdown site-wide on a hunch, measuring nothing, and discovering a year later that the only measurable outcome was subsidised scraping. The alternative is to run it as an actual experiment — a defined content subset, a pre-launch citation baseline, a control group, and a kill date. The IAB’s new AI-visibility guidance, published the same week, supplies the measurement floor: a minimum of 50 queries before citation data characterises anything. A markdown test measured on fewer is an anecdote either way.

    Time’s agent-facing ads deserve a separate verdict: genuinely interesting, genuinely unproven. If agents become a durable reading audience, someone will figure out how to monetise their attention — or their operators’ — and the publishers who ran early experiments will own the learning. But even Time has not shown advertiser results yet. That is R&D budget, not a revenue line, and it should be booked accordingly.

    03Why this matters for publishers

    The certain benefit accrues to the other sideThe 90% token saving lands on the AI company's compute bill. Your side of the trade — citation lift — is the side Promptwatch's 1.6M-citation analysis could not find.
    Machine-readability is a licensing asset, not a free featureIf your content being cheap to ingest is worth real money to AI companies, that value belongs in your licensing conversations, quoted and priced — not published as an open optimisation.
    Ungated markdown undermines your bot strategyIn the same week publishers compared notes on stealth crawlers routing around robots.txt, an open markdown endpoint hands every scraper — licensed or not — a cheaper way in.
    Nobody has measured the upside, including the pioneersGoogle says markdown is unnecessary; Time's agent ads have no advertiser results yet. Anyone selling you markdown as a proven visibility tactic is ahead of the evidence.

    04What publishers should do

    05The bottom line

    Markdown for AI bots is a rational-sounding tactic whose economics currently run backwards: the publisher bears the effort and surrenders the leverage, the AI company banks the savings, and the promised return is the one thing a 1.6-million-citation study failed to detect. Worth testing — with gates, baselines and a kill date. Not worth committing to. And if the machine audience is really coming, the right response is not to make yourself cheaper to read; it is to make cheap reading something the machines’ owners pay for.

    Sources & caveats

    Sources: Digiday Media Briefing, “The case for and against markdown for AI bots” (6 August 2026), which reports the ~90% token-reduction figure, Time’s practice of serving ads to AI agents inside its markdown files, Promptwatch’s analysis of 1.6 million citations finding markdown “did not increase” citation likelihood, and Google’s GEO guidance stating markdown is not necessary. The 50-query measurement minimum is from the IAB’s “Measuring Visibility in the AI Era” as reported by AdExchanger (3 August 2026). The token-saving percentage is an industry estimate rather than an audited figure; Time’s agent-facing ad results are, by the report’s account, not yet demonstrated.

    The weekly

    One letter a week, from the desk that runs the auctions.

    What actually moved in yield, CTV and curation across our publishers — written by the people who saw it, not a content team. No digests, no roundups, one email.

    One email a week. Unsubscribe in one click. We never share or sell the list.

    More from this issue

    Ran alongside this piece in the Weekly of 9 August 2026 — read the whole issue →