From 18523e5f78908e2a2a525defd6bcb66ae1c6d495 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Wed, 12 Jul 2023 13:56:35 -0700 Subject: [PATCH 01/13] Separate private-use from reserved --- spec/syntax.md | 43 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 8 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 1cedae38f4..69331e28fb 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -20,6 +20,7 @@ 1. [Patterns](#patterns) 1. [Expressions](#expressions) 1. [Reserved Sequences](#reserved) + 2. [Private-Use Sequences](#private-use) 1. [Tokens](#tokens) 1. [Keywords](#keywords) 1. [Text](#text) @@ -413,17 +414,13 @@ option = name [s] "=" [s] (literal / variable) #### Reserved **_Reserved_** annotations start with a reserved character -and are intended for future standardization -as well as private implementation use. +and are intended for future standardization. A _reserved_ _annotation_ MAY be empty or contain arbitrary text. This allows maximum flexibility in future standardization, -as future definitions are expected to define additional semantics and constraints +as future definitions MAY define additional semantics and constraints on the contents of these _annotations_. A _reserved_ _annotation_ does not include trailing whitespace. -Implementations MAY define their own meaning and semantics for -_reserved_ annotations that start with -the U+0026 AMPERSAND `&` or U+005E CIRCUMFLEX ACCENT `^` characters. Implementations MUST NOT assign meaning or semantics to an _annotation_ starting with `reserved-start`: these are reserved for future standardization. @@ -433,9 +430,8 @@ While a reserved sequence is technically "well-formed", unrecognized reserved sequences have no meaning and MAY result in errors during formatting. ```abnf -reserved = ( reserved-start / private-start ) reserved-body +reserved = reserved-start reserved-body reserved-start = "!" / "@" / "#" / "%" / "*" / "<" / ">" / "/" / "?" / "~" -private-start = "^" / "&" reserved-body = *( [s] 1*(reserved-char / reserved-escape / literal)) reserved-char = %x00-08 ; omit HTAB and LF / %x0B-0C ; omit CR @@ -446,6 +442,37 @@ reserved-char = %x00-08 ; omit HTAB and LF / %xE000-10FFFF ``` +#### Private-Use + +A **_private-use_** _annotation_ is an _annotation_ whose syntax is reserved +for use by a specific implementation or by private agreement between multiple +implementations. +Implementations MAY define their own meaning and semantics for _private-use_ annotations. + +A _private-use_ annotation starts with either U+0026 AMPERSAND `&` or U+005E CIRCUMFLEX ACCENT `^`. +The characters `\`, `{`, and `}` MUST be escaped as `\\`, `\{`, and `\}` when they +appear as part of a _private-use_ annotation. + +All other characters, including whitespace, are assigned meaning by the implementation. +A _private-use_ _annotation_ MAY be empty. + +**NOTE:** Users are cautioned that _private-use_ sequences cannot be reliably exchanged +and can result in errors during formatting. It is generally a better idea to use +the function registry to define additional formatting or annotation options. + +```abnf +private-use = private-start 1*(text) +private-start = "&" / "^" +``` + +> Here are some examples of what _private-use_ sequences might look like: +>> ``` +>> {Here's private use with an operand: {$foo &function}} +>> {Here's a placeholder that is entirely private-use: {&function anything here}} +>> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} +>> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} +>>``` + ## Tokens The grammar defines the following tokens for the purpose of the lexical analysis. From f11dbbc0997bfad952439c1e45e112b9813bfb21 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Wed, 12 Jul 2023 14:00:19 -0700 Subject: [PATCH 02/13] Implement private-use in the ABNF Note that I didn't address making reserved opaque. --- spec/message.abnf | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/spec/message.abnf b/spec/message.abnf index 1adfe7a3c9..9c8f36be96 100644 --- a/spec/message.abnf +++ b/spec/message.abnf @@ -42,11 +42,15 @@ unquoted = unquoted-start *name-char unquoted-start = name-start / DIGIT / "." / %xB7 / %x300-36F / %x203F-2040 -; reserve additional sigils for use by future versions of this -; specification or for private use by implementations -reserved = ( reserved-start / private-start ) reserved-body -reserved-start = "!" / "@" / "#" / "%" / "*" / "<" / ">" / "/" / "?" / "~" + +; reserve sigils for private-use by implementations +private-use = private-start 1*(text) private-start = "^" / "&" + +; reserve additional sigils for use by +; future versions of this specification +reserved = reserved-start reserved-body +reserved-start = "!" / "@" / "#" / "%" / "*" / "<" / ">" / "/" / "?" / "~" reserved-body = *( [s] 1*(reserved-char / reserved-escape / literal)) reserved-char = %x00-08 ; omit HTAB and LF / %x0B-0C ; omit CR From e0f70e9be997b84e9670f3ab249a00fc5c12b343 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Wed, 12 Jul 2023 14:01:14 -0700 Subject: [PATCH 03/13] "missed a spot" --- spec/message.abnf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/message.abnf b/spec/message.abnf index 9c8f36be96..54cf570cef 100644 --- a/spec/message.abnf +++ b/spec/message.abnf @@ -10,8 +10,8 @@ variant = when 1*(s key) [s] pattern key = literal / "*" expression = "{" [s] ((operand [s annotation]) / annotation) [s] "}" -operand = literal / variable -annotation = (function *(s option)) / reserved +operand = literal / variable +annotation = (function *(s option)) / reserved / private-use literal = quoted / unquoted variable = "$" name From 46fbf4be083f08909b0b4fb6204f6801d57be059 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Thu, 13 Jul 2023 09:16:43 -0700 Subject: [PATCH 04/13] Update spec/syntax.md Co-authored-by: Eemeli Aro --- spec/syntax.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/syntax.md b/spec/syntax.md index 69331e28fb..04403efee9 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -454,7 +454,7 @@ The characters `\`, `{`, and `}` MUST be escaped as `\\`, `\{`, and `\}` when th appear as part of a _private-use_ annotation. All other characters, including whitespace, are assigned meaning by the implementation. -A _private-use_ _annotation_ MAY be empty. +A _private-use_ _annotation_ MAY be empty after its first character. **NOTE:** Users are cautioned that _private-use_ sequences cannot be reliably exchanged and can result in errors during formatting. It is generally a better idea to use From a788820247be0a1c3a28a0732ab4b3c0925630db Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Thu, 13 Jul 2023 09:16:57 -0700 Subject: [PATCH 05/13] Update spec/syntax.md Co-authored-by: Eemeli Aro --- spec/syntax.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 04403efee9..15772577fb 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -457,8 +457,9 @@ All other characters, including whitespace, are assigned meaning by the implemen A _private-use_ _annotation_ MAY be empty after its first character. **NOTE:** Users are cautioned that _private-use_ sequences cannot be reliably exchanged -and can result in errors during formatting. It is generally a better idea to use -the function registry to define additional formatting or annotation options. +and can result in errors during formatting. +It is generally a better idea to use the function registry +to define additional formatting or annotation options. ```abnf private-use = private-start 1*(text) From 816866e92a7a98eda126a0eb3c2706e408620558 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Thu, 13 Jul 2023 09:32:49 -0700 Subject: [PATCH 06/13] Update spec/syntax.md Co-authored-by: Eemeli Aro --- spec/syntax.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 15772577fb..651d7745e3 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -445,8 +445,7 @@ reserved-char = %x00-08 ; omit HTAB and LF #### Private-Use A **_private-use_** _annotation_ is an _annotation_ whose syntax is reserved -for use by a specific implementation or by private agreement between multiple -implementations. +for use by a specific implementation or by private agreement between multiple implementations. Implementations MAY define their own meaning and semantics for _private-use_ annotations. A _private-use_ annotation starts with either U+0026 AMPERSAND `&` or U+005E CIRCUMFLEX ACCENT `^`. From 70d9c2766e9c05b916c910cdf232234702a60bca Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Thu, 13 Jul 2023 09:34:01 -0700 Subject: [PATCH 07/13] Update spec/syntax.md Co-authored-by: Eemeli Aro --- spec/syntax.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 651d7745e3..5362dc6563 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -467,8 +467,8 @@ private-start = "&" / "^" > Here are some examples of what _private-use_ sequences might look like: >> ``` ->> {Here's private use with an operand: {$foo &function}} ->> {Here's a placeholder that is entirely private-use: {&function anything here}} +>> {Here's private use with an operand: {$foo &bar}} +>> {Here's a placeholder that is entirely private-use: {&anything here}} >> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} >> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} >>``` From bbf4b864fdf22072057425cb7312c0449d2a9d83 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Fri, 14 Jul 2023 08:34:37 -0700 Subject: [PATCH 08/13] Update message.abnf --- spec/message.abnf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/message.abnf b/spec/message.abnf index 54cf570cef..b38bd2adea 100644 --- a/spec/message.abnf +++ b/spec/message.abnf @@ -44,7 +44,7 @@ unquoted-start = name-start / DIGIT / "." ; reserve sigils for private-use by implementations -private-use = private-start 1*(text) +private-use = private-start reserved-body private-start = "^" / "&" ; reserve additional sigils for use by From 2bb8d154517b38b1957d00b29d8d7ece8aaf46b7 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Fri, 14 Jul 2023 08:51:23 -0700 Subject: [PATCH 09/13] Address private-use and reserved escaping --- spec/syntax.md | 70 ++++++++++++++++++++++++++++---------------------- 1 file changed, 39 insertions(+), 31 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 5362dc6563..99bfce26f3 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -411,6 +411,45 @@ option = name [s] "=" [s] (literal / variable) > {{+h1 name=above-and-beyond}Above And Beyond{-h1}} > ``` +#### Private-Use + +A **_private-use_** _annotation_ is an _annotation_ whose syntax is reserved +for use by a specific implementation or by private agreement between multiple implementations. +Implementations MAY define their own meaning and semantics for _private-use_ annotations. + +A _private-use_ annotation starts with either U+0026 AMPERSAND `&` or U+005E CIRCUMFLEX ACCENT `^`. + +Characters, including whitespace, are assigned meaning by the implementation. +The characters `\`, `{`, and `}` MUST be escaped as `\\`, `\{`, and `\}` respectively +when they appear in the body of a _private-use_ annotation. +The character `|` is special: it SHOULD be escaped as `\|` in a _private-use_ annotation, +but can appear unescaped as long as it is paired with another `|`. This is an affordance to +allow _literals_ to appear in the private use syntax. +The definition of escapes in the `reserved-body` production is an affordance to +implementations that want to use a syntax exactly like other functions, in that it makes +_quoted_ _literals_. +A _private-use_ _annotation_ MAY be empty after its introducing sigil. + +**NOTE:** Users are cautioned that _private-use_ sequences cannot be reliably exchanged +and can result in errors during formatting. +It is generally a better idea to use the function registry +to define additional formatting or annotation options. + +```abnf +private-use = private-start reserved-body +private-start = "&" / "^" +``` + +> Here are some examples of what _private-use_ sequences might look like: +>> ``` +>> {Here's private use with an operand: {$foo &bar}} +>> {Here's a placeholder that is entirely private-use: {&anything here}} +>> {Here's a private-use function that uses normal function syntax: {$operand ^foo option=|literal|}} +>> {The character \| has to be paired or escaped: {&private || \| }} +>> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} +>> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} +>>``` + #### Reserved **_Reserved_** annotations start with a reserved character @@ -442,37 +481,6 @@ reserved-char = %x00-08 ; omit HTAB and LF / %xE000-10FFFF ``` -#### Private-Use - -A **_private-use_** _annotation_ is an _annotation_ whose syntax is reserved -for use by a specific implementation or by private agreement between multiple implementations. -Implementations MAY define their own meaning and semantics for _private-use_ annotations. - -A _private-use_ annotation starts with either U+0026 AMPERSAND `&` or U+005E CIRCUMFLEX ACCENT `^`. -The characters `\`, `{`, and `}` MUST be escaped as `\\`, `\{`, and `\}` when they -appear as part of a _private-use_ annotation. - -All other characters, including whitespace, are assigned meaning by the implementation. -A _private-use_ _annotation_ MAY be empty after its first character. - -**NOTE:** Users are cautioned that _private-use_ sequences cannot be reliably exchanged -and can result in errors during formatting. -It is generally a better idea to use the function registry -to define additional formatting or annotation options. - -```abnf -private-use = private-start 1*(text) -private-start = "&" / "^" -``` - -> Here are some examples of what _private-use_ sequences might look like: ->> ``` ->> {Here's private use with an operand: {$foo &bar}} ->> {Here's a placeholder that is entirely private-use: {&anything here}} ->> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} ->> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} ->>``` - ## Tokens The grammar defines the following tokens for the purpose of the lexical analysis. From 5228e9778fde36904df0ea1ee16dd63b24bea03e Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Fri, 14 Jul 2023 08:57:12 -0700 Subject: [PATCH 10/13] Update syntax.md --- spec/syntax.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 99bfce26f3..4aac849b30 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -19,8 +19,8 @@ 1. [Variants](#variants) 1. [Patterns](#patterns) 1. [Expressions](#expressions) - 1. [Reserved Sequences](#reserved) - 2. [Private-Use Sequences](#private-use) + 1. [Private-Use Sequences](#private-use) + 2. [Reserved Sequences](#reserved) 1. [Tokens](#tokens) 1. [Keywords](#keywords) 1. [Text](#text) @@ -420,14 +420,15 @@ Implementations MAY define their own meaning and semantics for _private-use_ ann A _private-use_ annotation starts with either U+0026 AMPERSAND `&` or U+005E CIRCUMFLEX ACCENT `^`. Characters, including whitespace, are assigned meaning by the implementation. -The characters `\`, `{`, and `}` MUST be escaped as `\\`, `\{`, and `\}` respectively +The definition of escapes in the `reserved-body` production, used for the body of +a _private-use_ annotation is an affordance to implementations that +wish to use a syntax exactly like other functions. Specifically: +* The characters `\`, `{`, and `}` MUST be escaped as `\\`, `\{`, and `\}` respectively when they appear in the body of a _private-use_ annotation. -The character `|` is special: it SHOULD be escaped as `\|` in a _private-use_ annotation, +* The character `|` is special: it SHOULD be escaped as `\|` in a _private-use_ annotation, but can appear unescaped as long as it is paired with another `|`. This is an affordance to allow _literals_ to appear in the private use syntax. -The definition of escapes in the `reserved-body` production is an affordance to -implementations that want to use a syntax exactly like other functions, in that it makes -_quoted_ _literals_. + A _private-use_ _annotation_ MAY be empty after its introducing sigil. **NOTE:** Users are cautioned that _private-use_ sequences cannot be reliably exchanged @@ -445,7 +446,7 @@ private-start = "&" / "^" >> {Here's private use with an operand: {$foo &bar}} >> {Here's a placeholder that is entirely private-use: {&anything here}} >> {Here's a private-use function that uses normal function syntax: {$operand ^foo option=|literal|}} ->> {The character \| has to be paired or escaped: {&private || \| }} +>> {The character \| has to be paired or escaped: {&private || |something between| or isolated: \| }} >> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} >> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} >>``` From 77ef04eee1f3b79ed2809d5eaa25c36b2e54df0d Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Sat, 15 Jul 2023 15:51:45 -0700 Subject: [PATCH 11/13] Address changes to private use in formatting.md --- spec/formatting.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/spec/formatting.md b/spec/formatting.md index 6cc866a688..f9724062ba 100644 --- a/spec/formatting.md +++ b/spec/formatting.md @@ -92,7 +92,7 @@ and different implementations MAY choose to perform different levels of resoluti > or some other locally appropriate value. Depending on the presence or absence of an _operand_ -and a _function_ or _reserved_ _annotation_, +and a _function_, _private-use_, or _reserved_ _annotation_, one of the following is used to resolve the value of the _expression_: - If the _expression_ contains no _annotation_, @@ -100,10 +100,12 @@ one of the following is used to resolve the value of the _expression_: depending on the shape of the _operand_. - Else, if the _expression_ has a _function_ _annotation_, its resolved value is defined by _function resolution_. -- Else, the _expression_ has a _reserved_ _annotation_. - If the _annotation_ uses a `private-start` character that the implementation supports, - its value is resolved according to the implementation's specification. - Else, an Unsupported Expression error is emitted and a fallback value is used as its value. +- Else, if the _expression_ has a _private-use_ _annotation_, + its resolved value is defined according to the implementations's specification. +- Else, if the _expression_ has a _reserved_ _annotation_, + an Unsupported Expression error is emitted and a fallback value is used as its value. + Else the contents are syntactically invalid and + an Unsupported Expression error is emitted and a fallback value is used as its value. ### Literal Resolution @@ -183,6 +185,8 @@ An _expression_ fails to resolve when: - A _variable_ _operand_ fails to resolve. - A _function_ _annotation_ fails to resolve. +- A _private-use_ _annotation_ is unsupported by the implementation or if + a _private-use_ _annotation_ fails to resolve. - The _expression_ has a _reserved_ _annotation_. The _fallback value_ depends on the contents of the _expression_: From 598d95e565770be80cb8274999b3c992543c117f Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Mon, 17 Jul 2023 11:03:22 -0700 Subject: [PATCH 12/13] Fix "double-indention" of one example I missed this earlier. --- spec/syntax.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/spec/syntax.md b/spec/syntax.md index 4aac849b30..ab8e9bf8f6 100644 --- a/spec/syntax.md +++ b/spec/syntax.md @@ -442,14 +442,14 @@ private-start = "&" / "^" ``` > Here are some examples of what _private-use_ sequences might look like: ->> ``` ->> {Here's private use with an operand: {$foo &bar}} ->> {Here's a placeholder that is entirely private-use: {&anything here}} ->> {Here's a private-use function that uses normal function syntax: {$operand ^foo option=|literal|}} ->> {The character \| has to be paired or escaped: {&private || |something between| or isolated: \| }} ->> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} ->> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} ->>``` +> ``` +> {Here's private use with an operand: {$foo &bar}} +> {Here's a placeholder that is entirely private-use: {&anything here}} +> {Here's a private-use function that uses normal function syntax: {$operand ^foo option=|literal|}} +> {The character \| has to be paired or escaped: {&private || |something between| or isolated: \| }} +> {Stop {& "translate 'stop' as a verb" might be a translator instruction or comment }} +> {Protect stuff in {^ph}{^/ph}private use{^ph}{^/ph}} +>``` #### Reserved From 81be51b1cafe5f26d63424efdb3847bac3924cd2 Mon Sep 17 00:00:00 2001 From: Addison Phillips Date: Mon, 17 Jul 2023 11:04:45 -0700 Subject: [PATCH 13/13] Remove final 'else' --- spec/formatting.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/spec/formatting.md b/spec/formatting.md index f9724062ba..15d28f9c8c 100644 --- a/spec/formatting.md +++ b/spec/formatting.md @@ -102,9 +102,7 @@ one of the following is used to resolve the value of the _expression_: its resolved value is defined by _function resolution_. - Else, if the _expression_ has a _private-use_ _annotation_, its resolved value is defined according to the implementations's specification. -- Else, if the _expression_ has a _reserved_ _annotation_, - an Unsupported Expression error is emitted and a fallback value is used as its value. - Else the contents are syntactically invalid and +- Else, the _expression_ has a _reserved_ _annotation_, an Unsupported Expression error is emitted and a fallback value is used as its value. ### Literal Resolution