The single most operational decision a publisher faces in 2026 — let AI crawlers in, or shut them out — got both a rulebook and a counter-argument this week. IAB Tech Lab released bot- and crawler-management guidance for public comment (open through June 26), and almost in the same breath, Microsoft told publishers the opposite of what their instincts are screaming: don’t block the AI bots.
01What IAB Tech Lab put on the table
The new guidance, released for public comment on May 27, outlines the range of approaches content owners can take to inform and control access from non-human user agents — the AI crawlers, scrapers, and agents now hammering publisher infrastructure. It’s designed to be adopted alongside the CoMP API V1 (Content Monetization Protocol), giving publishers a standardized, machine-readable way to declare who may access content, under what terms, and for what use — training, retrieval, agentic answering, and so on.
The point is to move publishers past the blunt, binary world of robots.txt — where your only real options are “allow everything” or “block and hope” — toward granular, enforceable, and auditable access signals that an AI company can’t credibly claim it didn’t see. That “couldn’t have known” defense, notably, is exactly what CNN is litigating against Perplexity this week.
02Microsoft’s contrarian pitch
At the same time, Microsoft is urging publishers not to block AI bots — arguing that visibility inside AI answers and assistants is becoming a primary discovery channel, and that publishers who wall themselves off will simply vanish from where audiences increasingly are. It’s a self-interested argument from a company building answer products, but it’s not wrong on the facts: blocking protects content but can accelerate irrelevance.
This is the real publisher dilemma in one frame. Block, and you protect your content but disappear from the AI surfaces where discovery is migrating. Allow, and you fuel the engines cannibalizing your traffic — for little or no pay. The binary is the trap.
03Why the standards matter
The IAB’s bot-management guidance and CoMP API exist precisely to dissolve that binary. Instead of “block vs. allow,” publishers get a third path: conditional, priced, auditable access. Let the bots in — but on declared terms, with a standard the whole ecosystem (and eventually the courts and regulators) recognizes. That’s what turns “you scraped us” from a he-said/she-said into a documented violation, and what turns “let us crawl you” into a licensing conversation with a meter on it.
04Why this matters
| Leverage | A standardized access-and-consent signal is the technical backbone of every AI licensing deal. Without it, you're negotiating on vibes. |
|---|---|
| Litigation posture | Clear, machine-readable access terms strengthen any future copyright or unauthorized-use claim — and weaken the "we didn't know" defense. |
| Discovery vs. cannibalization | The Microsoft argument is real. You may want certain surfaces (marketing, top-funnel, evergreen) crawled for visibility while walling off premium journalism. Standards let you make that split deliberately instead of all-or-nothing. |
05What publishers should do this quarter
06What marketers should do
07The bottom line
“Block or don’t block” is the wrong question. The right question is on what terms — and this week the industry finally started building the plumbing to answer it. Publishers who engage with the IAB’s guidance now will be the ones setting the price of access, rather than discovering after the fact that they gave it away.