AI Preferences T. Vaughan Internet-Draft Common Crawl Foundation Intended status: Informational 20 January 2025 Expires: 24 July 2025 Vocabulary for Expressing Content Preferences for AI Training draft-vaughan-aipref-vocab-00 Abstract This document proposes a vocabulary for expressing content preferences for rightsholders who wish to manage the use of their content in AI training. This vocabulary allows publishers to express preferences through metadata or content-delivery protocols. The vocabulary can be applied at different levels of granularity and incorporates preferences for permissions, usage scope, and data retention, providing a foundation for interoperability across various Internet protocols. About This Document This note is to be removed before publishing as an RFC. The latest revision of this draft can be found at https://thunderpoot.github.io/draft-vaughan-aipref-vocab/draft- vaughan-aipref-vocab.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-vaughan-aipref- vocab/. Discussion of this document takes place on the AI Preferences mailing list (mailto:ai-control@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/ai-control/. Subscribe at https://www.ietf.org/mailman/listinfo/ai-control/. Source for this draft and an issue tracker can be found at https://github.com/thunderpoot/draft-vaughan-aipref-vocab. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Vaughan Expires 24 July 2025 [Page 1] Internet-Draft AIPREF Vocab January 2025 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 24 July 2025. Copyright Notice Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Conventions and Definitions . . . . . . . . . . . . . . . . . 3 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. Vocabulary Elements / Preference Signals . . . . . . . . . . 3 4.1. Permission . . . . . . . . . . . . . . . . . . . . . . . 3 4.2. Purpose . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.3. Temporal Restrictions . . . . . . . . . . . . . . . . . . 4 4.4. Content-Specific Granularity . . . . . . . . . . . . . . 4 4.5. Content Type . . . . . . . . . . . . . . . . . . . . . . 4 4.6. Derivative Content . . . . . . . . . . . . . . . . . . . 4 4.7. Data Retention . . . . . . . . . . . . . . . . . . . . . 5 4.8. Preference Persistence . . . . . . . . . . . . . . . . . 5 4.9. Precedence . . . . . . . . . . . . . . . . . . . . . . . 5 4.10. Geographic Restrictions . . . . . . . . . . . . . . . . . 5 5. Implementation Considerations . . . . . . . . . . . . . . . . 5 5.1. HTTP Headers . . . . . . . . . . . . . . . . . . . . . . 6 5.2. Robots Exclusion Protocol (REP) . . . . . . . . . . . . . 6 5.3. Tags for Sub-Document Level Control . . . . . . . 6 5.4. “Well-Known” Locations . . . . . . . . . . . . . . . . . 7 5.5. Embedded Metadata . . . . . . . . . . . . . . . . . . . . 9 5.6. Content Credentials (ISO 22144) . . . . . . . . . . . . . 10 5.7. ISCC (ISO 24138) . . . . . . . . . . . . . . . . . . . . 10 6. Example Usage Scenarios . . . . . . . . . . . . . . . . . . . 10 7. Security Considerations . . . . . . . . . . . . . . . . . . . 10 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 Vaughan Expires 24 July 2025 [Page 2] Internet-Draft AIPREF Vocab January 2025 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 9.1. Normative References . . . . . . . . . . . . . . . . . . 10 9.2. Informative References . . . . . . . . . . . . . . . . . 11 Appendix A. Table of Preference Signals . . . . . . . . . . . . 11 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 13 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 1. Introduction As AI models become more reliant on large-scale data (driven by scaling laws that link model performance to dataset size), content publishers seek ways to control how their content is used in training these models. This draft provides a vocabulary that enables publishers to signal preferences for AI training concerning their content. 2. Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 3. Scope The AI-PREF vocabulary is limited to expressing content preferences for AI training and does not include enforcement mechanisms or client authentication. Default opt-in or opt-out statuses are beyond the scope of this proposal, as it focuses solely on establishing a standard for signalling explicit preferences. In cases where no preferences are signalled, the decision on whether this constitutes an opt-in or opt-out should be determined at the policy level downstream. It is important to note that preference signals are advisory. 4. Vocabulary Elements / Preference Signals 4.1. Permission Basic indicators of whether content can be used for AI training. * *allow_training*: Boolean * *restricted_training:* public, non-commercial, internal, licensed Vaughan Expires 24 July 2025 [Page 3] Internet-Draft AIPREF Vocab January 2025 4.2. Purpose Defines acceptable uses in training. * *purpose*: String: - *generation*: Creating models that are capable of generating content - *embedding*: Converting content to vector representations o *classification*: Categorising or labelling content o *summary*: Creating condensed versions of content o *paraphrase*: Creating derivative versions of content o *quotation*: Repetition of a passage or fragment of original content o *translation*: Converting content between languages 4.3. Temporal Restrictions Specifies the date range for training use. * *effective_date*: ISO 8601 Date string * *expiration_date*: ISO 8601 Date string 4.4. Content-Specific Granularity Defines the scope of applicability. Refers to the level at which preferences apply within the content. * *scope*: global, content-specific, conditional 4.5. Content Type Specifies content types the preference applies to. * *mime_type*: text, image, video, audio, application, [RFC2046]. 4.6. Derivative Content Allows or restricts derivatives like summaries. * *allow_derivatives*: Boolean Vaughan Expires 24 July 2025 [Page 4] Internet-Draft AIPREF Vocab January 2025 * *derivative_type*: summary, paraphrase, translation 4.7. Data Retention Defines content retention period post-training. * *retention_period*: ISO 8601 Duration string (e.g., P3Y6M4DT12H30M5S) 4.8. Preference Persistence Indicates if preferences should persist in derived datasets, or be optional. A derived dataset is the result of processing, transforming, or extracting information from the original source, such as aggregated statistics and summaries, or subsets of data. * *metadata_persistence*: Boolean 4.9. Precedence Conflicts should be resolved by assigning precedence values (e.g., high, medium, low) to rules, with a defined hierarchy that allows content producers to override publishers, domain operators, and others as necessary. * *precedence:* Sets priority when preferences conflict with other layered preferences. 4.10. Geographic Restrictions Specifies regions where preferences apply, ISO 3166-1. * *geo_limitations:* Specifies geographic regions where training permissions apply. 5. Implementation Considerations Implementing the AI-PREF vocabulary effectively can be accomplished using various mechanisms, depending on the needs and existing infrastructure of content publishers. Approaches include, but are not limited to, using HTTP headers, possible extensions to [RFC9309] ([PURPOSE]), and (for example) tags and other embedded data (such as EXIF) for sub-document-level control. Vaughan Expires 24 July 2025 [Page 5] Internet-Draft AIPREF Vocab January 2025 5.1. HTTP Headers Publishers can use HTTP headers to communicate AI-PREF preferences directly in response to client requests. This approach allows fine- grained control and easy integration into existing server configurations. *Example header:* AI-PREF: allow_training=true; purpose=generation,classification; retention_period=P3Y6M4DT12H30M5S This header specifies that the content can be used for text generation and classification, with a retention period of 3 years, 6 months, 4 days, 12 hours, 30 minutes, and 5 seconds. The syntax and options should be carefully chosen to ensure compatibility with common web servers and clients. 5.2. Robots Exclusion Protocol (REP) For publishers who already use REP (as defined in RFC9309 (https://datatracker.ietf.org/doc/rfc9309/)), extending REP rules to include AI-PREF preferences could be beneficial. Example rule: User-agent: * Allow-Training: non-commercial Purpose: embedding, summarisation This REP rule specifies that all user agents are allowed to use the content for non-commercial AI training, limited to embedding and summarisation purposes. Further extensions to REP could specify additional constraints, such as geographic limitations or temporal restrictions. 5.3. Tags for Sub-Document Level Control To specify AI-PREF preferences at the level of individual HTML documents or specific parts of a document, tags and HTML attributes can be used. Example tag: Example HTML attribute:
Vaughan Expires 24 July 2025 [Page 6] Internet-Draft AIPREF Vocab January 2025 The methods above specify that AI training is not allowed for the content of this document, with no retention period permitted. tags can be used to provide specific content preferences for a specific piece of content, and thus provide a flexible way to manage AI training signals at a more granular level. 5.4. “Well-Known” Locations According to [RFC8615], “well-known” locations can serve metadata or configuration information that is easily discoverable by automated clients. AI-PREF preferences can be published at a “well-known” URL. There is already the Text and Data Mining Reservation Protocol (TDMRep (https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep- 20240510/)) which has the same or overlapping intent. Example: https://example.com/.well-known/aipref At this URL, a JSON or other structured format can specify AI-PREF preferences for the entire domain or specific content types. Example JSON 1: { "allow_training": false, "purpose": ["generation"], "retention_period": "0" } Example JSON 2: Vaughan Expires 24 July 2025 [Page 7] Internet-Draft AIPREF Vocab January 2025 { "version": "1.0", "resources": [ { "path": "/videos/tutorial.mp4", "type": "video/mp4", "components": [ { "name": "Introduction", "time-range": "00:00:00-00:01:00", "preferences": { "classification": "allowed", "embedding": "allowed" } }, { "name": "Main Content", "time-range": "00:01:01-00:05:00", "preferences": { "generation": "prohibited", "summarization": "allowed" } } ] } ] } This approach simplifies discovery for automated clients and provides a centralised way to communicate content preferences across a domain. TDMRep Example: A rightsholder could expose a “well-known” TDMRep file at: https://example.com/.well-known/tdmrep Example TDMRep JSON Content: Vaughan Expires 24 July 2025 [Page 8] Internet-Draft AIPREF Vocab January 2025 { "version": "1.0", "license": "https://example.com/license", "contact": { "email": "tdm-support@example.com", "url": "https://example.com/contact" }, "resources": [ { "path": "/articles/", "type": "text/html", "restriction": "no-crawling" }, { "path": "/api/data/", "type": "application/json", "restriction": "license-required" } ] } 5.5. Embedded Metadata Preferences for multimodal data can be embedded directly into file metadata (such as EXIF or XMP) as self-contained control signals. Compatibility and tamper resistance (e.g. signing) should be considered. Example EXIF: AI-Pref-Allow-Training: false AI-Pref-Purpose: embedding AI-Pref-Retention-Period: 0 Example PDF Metadata Using XMP: Text mining allowed; Data sharing restricted Preferences can be applied at the file level, or even to specific components (e.g., chapters in a PDF or frames in a video). Vaughan Expires 24 July 2025 [Page 9] Internet-Draft AIPREF Vocab January 2025 Example WEBVTT: WEBVTT 00:00:00.000 --> 00:01:00.000 Usage Preferences: allow_training=true; purpose=generation,classification 00:01:01.000 --> 00:05:00.000 Usage Preferences: allow_training=false; 5.6. Content Credentials (ISO 22144) TBD 5.7. ISCC (ISO 24138) TBD 6. Example Usage Scenarios TODO examples 7. Security Considerations This document does not affect the security of the Internet. AI-PREF preferences do not include enforcement mechanisms, which should be addressed by AI model developers. Publishers should be aware that preferences may not prevent unauthorised use and may rely on mutual agreements or legal protections. 8. IANA Considerations This document does not require any immediate IANA actions but may suggest future registry entries for the vocabulary terms to support interoperability. 9. References 9.1. Normative References [RFC2046] Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, DOI 10.17487/RFC2046, November 1996, . Vaughan Expires 24 July 2025 [Page 10] Internet-Draft AIPREF Vocab January 2025 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8615] Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, DOI 10.17487/RFC8615, May 2019, . [RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, September 2022, . 9.2. Informative References [PURPOSE] Illyes, G., "Robots Exclusion Protocol User Agent Purpose Extension", Work in Progress, Internet-Draft, draft- illyes-rep-purpose-00, 18 October 2024, . Appendix A. Table of Preference Signals This table defines terms and values that specify metadata preferences for the use of content in AI training. Each term includes a description of its purpose and example values: +======================+===============+=================+==========================+ |Term |Values |Description |Example | +======================+===============+=================+==========================+ |allow_training |Boolean |Basic indicator |allow_training: false | | | |of whether | | | | |content can be | | | | |used for AI | | | | |training | | +----------------------+---------------+-----------------+--------------------------+ |purpose |String: |Defines |purpose: classification, | | |generation, |acceptable |summarisation | | |classification,|applications for | | | |summarisation, |training e.g. | | | |embedding, etc |fine-tuning, | | | | |classification, | | | | |summarisation, | | Vaughan Expires 24 July 2025 [Page 11] Internet-Draft AIPREF Vocab January 2025 | | |etc | | +----------------------+---------------+-----------------+--------------------------+ |effective_date |Date string, |Start date of |effective_date: | | |*ISO 8601* |when permissions |2024-10-30T15:52:55.440238| | | |take effect | | +----------------------+---------------+-----------------+--------------------------+ |expiration_date |Date string, |Date after which |expiration_date: | | |*ISO 8601* |permissions no |2024-10-30T15:52:55.440238| | | |longer apply | | +----------------------+---------------+-----------------+--------------------------+ |scope |String: global,|Defines whether |scope: content-specific | | |content- |the preferences | | | |specific, |apply | | | |conditional |universally, to | | | | |specific content,| | | | |or under certain | | | | |conditions | | +----------------------+---------------+-----------------+--------------------------+ |mime_type |text, image, |Specifies the |mime_type: text, image | | |video, audio |type(s) of | | | | |content the | | | | |preference | | | | |applies to | | +----------------------+---------------+-----------------+--------------------------+ |allow_derivatives |Boolean |Indicates whether|allow_derivatives: true | | | |derivative works | | | | |(summaries, | | | | |paraphrasing) are| | | | |allowed based on | | | | |content | | +----------------------+---------------+-----------------+--------------------------+ |derivative_type |String: |Lists permissible|derivative_type: summary, | | |summary, |types is |paraphrase | | |paraphrase, |allow_derivatives| | | |translation |is true | | +----------------------+---------------+-----------------+--------------------------+ |retention_period |Duration |Specifies how |P3Y6M4DT12H30M5S | | |string, *ISO |long content may |representing three years, | | |8601* |be retained after|six months, four days, | | | |use (e.g. after |twelve hours, thirty | | | |training). |minutes, and five seconds.| +----------------------+---------------+-----------------+--------------------------+ |preference_persistence|Boolean |Whether |preference_persistence: | | | |preferences must |true | | | |persist with | | | | |derived data, | | | | |boolean for | | | | |either required | | Vaughan Expires 24 July 2025 [Page 12] Internet-Draft AIPREF Vocab January 2025 | | |or optional | | +----------------------+---------------+-----------------+--------------------------+ |precedence |String:high, |Sets priority |precedence: high | | |medium, low |when preferences | | | | |conflict with | | | | |other layered | | | | |preferences | | +----------------------+---------------+-----------------+--------------------------+ |geo_limitations |Location codes,|Specifies |geo_limitations: EU, US | | |*ISO 3166* |geographic | | | | |regions where | | | | |training | | | | |permissions apply| | +----------------------+---------------+-----------------+--------------------------+ Table 1 Acknowledgments * Greg Lindahl * Sebastian Nagel * Gary Illyes * Mark Nottingham * Suresh Krishnan * Martin Thomson * Paul Keller * Leonard Rosenthol * Special thanks to the program committee and contributing members of the IAB AI-CONTROL Workshop, and aipref Working Group. Author's Address Thom Vaughan Common Crawl Foundation Email: thom@commoncrawl.org Vaughan Expires 24 July 2025 [Page 13]