--- title: "IETF's AI Preferences group is still drafting, not shipping" url: https://surfil.com/news/ietf-aipref-robots-txt-successor section: news language: en published: 2026-08-18T00:00:00.000Z modified: 2026-08-18T00:00:00.000Z author: Adam Hafez topics: ["Technical SEO", "AI search"] --- # IETF's AI Preferences group is still drafting, not shipping ## The short answer The IETF chartered an AI Preferences (AIPREF) working group on 7 January 2025 to standardize how sites express preferences about AI training and use. As of September 2026 it has two Internet-Drafts, not published RFCs, covering a vocabulary of preference terms and how to attach them through robots.txt or an HTTP header. Nothing here changes today's robots.txt configuration. ## Key takeaways - The IETF's AI Preferences working group was chartered on 7 January 2025. - The charter names the Robots Exclusion Protocol, RFC 9309, and HTTP response headers as candidate delivery mechanisms rather than proposing to replace either. - As of September 2026 the group has produced two Internet-Drafts, not published RFCs, a vocabulary draft and an attachment draft, both on the Proposed Standard track. - The vocabulary draft, in version 08 of September 2026, defines three usage categories, train-ai, ai-use and search, each set to allow or disallow or left unknown. - The attachment draft proposes a Content-Usage directive in robots.txt alongside a matching Content-Usage HTTP response header, not a replacement file. The IETF has a working group standardizing how a site tells AI systems what they may do with its content, and it has been at it for a year and a half without publishing a single RFC. The AI Preferences (AIPREF) working group was chartered on 7 January 2025. As of September 2026 it has produced two Internet-Drafts. Neither is a finished standard. ## What the charter actually asks for AIPREF's charter describes the goal as standardizing "building blocks that allow for the expression of preferences about how content is collected and processed for Artificial Intelligence (AI) model development, deployment, and use." It names Well-Known URIs under RFC 8615, specifically the Robots Exclusion Protocol of RFC 9309, and HTTP response header fields as candidate mechanisms for attaching those preferences to content. The charter frames these as delivery options the working group can build on, not as a problem statement about robots.txt itself, and the datatracker record does not contain language calling existing AI-vendor signals confusing or non-standard. That framing shows up in commentary around the group, not in the charter text. The practical background is not controversial even without a quotable charter line: several AI companies already publish their own robots.txt user-agent tokens, GPTBot and Google-Extended among them, and site owners have had no single vocabulary for saying yes to search use while saying no to model training. AIPREF's job is to give that distinction a standard shape. This is a separate effort from llms.txt, the informal file-at-the-root convention some blog posts have wrongly attributed to a W3C standards track; AIPREF never touches llms.txt and has no relationship to it. ## The two drafts, and what they cover **draft-ietf-aipref-vocab** defines the preference vocabulary. As of version 08, dated 14 September 2026, it specifies three usage categories: train-ai, for using content to modify a generative AI model's learned parameters; ai-use, for using content as input to a generative AI model; and search, for applications whose primary purpose is directing users to the original source. Version 07, in August, had only train-ai and search. A category takes allow or disallow, and one with no stated preference is unknown; the draft takes no position on a default. **draft-ietf-aipref-attach**, version 05, also dated 18 August 2026, defines how a preference gets attached to content: a new Content-Usage directive inside robots.txt, and a matching Content-Usage HTTP response header for sites that would rather signal at the response level than the file level. Both mechanisms are meant to coexist with RFC 9309 rather than supersede it. The vocabulary draft opens with a note that its contents do not reflect working group consensus in whole or in part. The vocabulary draft expires on 18 March 2027 and the attachment draft on 20 February 2027 if not revised or advanced. That is normal for an active draft, not a sign of stalling, but it does mean the vocabulary and the syntax can still change before either document reaches working group last call. ## Why we care Nothing here changes what belongs in a robots.txt file today. The working group has not published an RFC, the vocabulary covers just three usage categories, and the attachment syntax is still draft text that IESG review could still alter. What's worth tracking is the direction: if AIPREF finishes, sites get one IETF-sanctioned way to say "index and cite me, don't train on me" instead of the patchwork of vendor-specific tokens crawlers currently interpret however they choose. Technical SEOs should watch draft-ietf-aipref-attach advance past working group last call before treating a Content-Usage directive as something to deploy, not before. ## Common questions ### Does this replace robots.txt? No. The attachment draft adds a new Content-Usage directive that would sit inside an existing robots.txt file, alongside a matching HTTP response header for sites that prefer that route. RFC 9309 stays the base protocol either way. ### Can I use AIPREF today? Not as a ratified standard. Both documents are Internet-Drafts, meaning they can still change before working group last call, IESG review and publication as RFCs. Nothing in this effort is a shipped feature any crawler is contractually bound to honor yet. ### Who is actually building this? The IETF AI Preferences working group, chaired by Suresh Krishnan and Mark Nottingham, with liaison relationships to the IPTC, the PLUS Coalition, and WHATWG or W3C for related content-format work. ## Sources 1. [AI Preferences (aipref) - Working Group](https://datatracker.ietf.org/wg/aipref/about/) - IETF Datatracker (primary) 2. [Charter for Working Group: AI Preferences (aipref)](https://datatracker.ietf.org/doc/charter-ietf-aipref/) - IETF Datatracker (primary) 3. [A Vocabulary For Expressing AI Usage Preferences](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/) - IETF Datatracker 4. [Associating AI Usage Preferences with Content in HTTP](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/) - IETF Datatracker 5. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) - RFC Editor