From @gibson042:

I've asked before, and will repeat the question: what is the value of staying close to XML Nmtoken but not matching it exactly? If there's going to be deviation (and unexplained deviation at that), what makes that a better foundation than e.g. UAX 31 Identifiers, or for that matter of dropping the pretense that this syntax has any relevant relationship to another one?

UAX 31 identifiers support a much more limited set of strings than we want to support as unquoted values; for example values such as 42, 13.0 and html:b are currently valid unquoted literals, but are not valid UAX 31 identifiers. XML Nmtoken is really close to being exactly what we want, with two specific starting-character exclusions we've needed to carve out.

One key differentiator here is that for unquoted we're not trying to match an "identifier" grammar, but a "token" grammar. As far as I'm aware, no real regularity exists on this, and the XML Nmtoken seems like a pretty good starting point.

From @peter-b:

We really do not need a N+1th incompatible specification for identifiers. It would be greatly preferable to either:

• use an identifier grammar production that exactly matches some existing standard
• use UAX 31

Any other choice requires an extraordinarily strong justification, which has not been provided here.

Please correct me if I'm mistaken, but this seems like a point that's more about name rather than unquoted, yes? If this is the case, replies to the following should be spun off into their own issue rather than being tracked under this PR:

For name, using UAX 31 identifiers would be conceivable as a starting point, provided that the grammar also allowed for a path-splitting character such as ., -, or : to support constructs like html:b or foo.bar. This is not an uncontroversial topic, and when settling on a syntax the WG opted to allow each implementation to figure out for itself e.g. whether the . in foo.bar has any special meaning, rather than making the call at the syntax level. Basing name on XML Name supports this, while other choices would require this discussion to be reopened.

XML Nmtoken is really close to being exactly what we want, with two specific starting-character exclusions we've needed to carve out.

"Needed" seems like an overly strong characterization. What problems would ensue from updating the syntax to match XML Nmtoken exactly (keeping in mind that it is possible for source to be syntactically valid while still being rejected for higher-level semantic reasons)? And if the deviation is actually necessary, shouldn't the reason be documented in a grammar comment, either directly or with a referencing link?

What problems would ensue from updating the syntax to match XML Nmtoken exactly (keeping in mind that it is possible for source to be syntactically valid while still being rejected for higher-level semantic reasons)?

XML Nmtoken may start with either the : or - characters, which are function start indicators in MF2. This causes an overlap at the beginning of an expression, which may have an operand. Effectively, matching the rule exactly for unquoted would require us to change the function start characters, or disallow unquoted as an operand; only allowing it as an option value.

And if the deviation is actually necessary, shouldn't the reason be documented in a grammar comment, either directly or with a referencing link?

This clarification is provided in the syntax.md, to which this PR adds this text:

To make _unquoted_ literals distinct from _function_ names,
a literal MUST be quoted if it begins with a `:`
or if it begins with a `-` that is **not** followed by a `.` or a digit.

At least my understanding is that our syntax is defined by both the message.abnf as well as the syntax.md, and that duplication of the latter into the former would not necessarily be beneficial -- if you're looking for explanations, then syntax.md already provides that, along with the ABNF rules.

XML Nmtoken may start with either the : or - characters, which are function start indicators in MF2. This causes an overlap at the beginning of an expression, which may have an operand. Effectively, matching the rule exactly for unquoted would require us to change the function start characters, or disallow unquoted as an operand; only allowing it as an option value.

If XML Nmtoken is sufficiently important to use as a foundation for unquoted literals, then having overlap between it and function prefix sigils seems like an unnecessary misstep. Without dragging this too far into the weeds, I will note that most ASCII punctuation characters are excluded from NameChar and therefore available for that use in MF2, particularly all of those that are currently included in reserved-start = "!" / "@" / "#" / "%" / "^" / "&" / "*" / "<" / ">" / "?" / "~". Accordingly, of the three PRs relating to unquoted negative number operands, I think {#open}...{/close} of #398 heads in the best direction (although it would still need to replace {:functionName} with something like {!functionName} so that valid Nmtoken instances like ":functionName" can be used as unquoted operands).

@gibson042 That's a good point. If we want to claim that our naming is rooted in XML, why not just go all the way and use Name and Nmtoken and choose sigils (there are plenty available) that don't conflict with these? The main thing is that we'd need to change our (arbitrary) choice of : too. It would greatly simplify the logic behind our naming and perhaps our syntax.

The important issue is that our syntax be even as natural and non-error prone as possible. I don't see that there is much if any value to adhering strictly to Nmtoken if that adherence compromises the syntax, as long as we have an unambiguous definition.

Sure, but are you claiming that adopting Nmtoken would compromise the syntax? Is {:foo} really more natural than {!foo}/{@foo}/{#foo}/etc.? Is it really less error prone to accept {-1 :bar} while rejecting {+1 :bar}?

@macchiati I agree. However, if we can achieve full compatibility with our chosen namespace scheme by choosing a different (somewhat randomly chosen) sigil, doesn't that (slightly) reduce potential errors? The syntax is only required to be internally consistent, but its better if it is also widely compatible with the contexts in which it will appear and if exterior software requires only minimal modification to interact with our constructs.

Of course, it isn't as if the world is constructed around Nmtoken. It's nice that our naming matches something, but that something is not exactly like most programming language namespaces and it will conflict with naming conventions established in some other resource, templating, scripting, or programming contexts somewhere. As a reminder, the reason we chose Nmtoken (and Name) was to maximize compatibility with (potential) LDML constructs, since CLDR uses XML. The "carve out" for various sigils doesn't conflict with naming in LDML currently and future conflicts are under our control. If we don't care about other XML dialects, non-conformance with Nmtoken (et al) may ultimately not matter. Most users will name their variables and functions something rational (probably an ASCII alphanumeric token).

I continue to believe that this PR treats the symptom rather than the root cause. I think there are two blockers before we can consider this PR for merging:

• Finish the design of the open/close spannables in Design document for Open/Close Expressions #470, including their syntax.
• Document the use-cases for unquoted literals in Document why we need unquoted literals #478.

Just to note, on the call of 4 September we agreed to merge this on or immediately after the end of the Seville colloquium on 13 September, unless we explicitly decide against doing so. No such decision has been made.

I agree that this is treating the symptom rather than the root cause, but that something very much like this is required for negative number support with our current syntax. We should patch this now, while acknowledging that it may be wholly superseded by later changes.

That agreement was based on the assumption that we would have completed the design of open/close in Seville. That didn't happen and the design is still ongoing in #470.

I want to this PR to stay open because I want this to be a friction point, so that we don't get complacent with the current proposed design on open/close spannables.

An alternative which I'd be happy to go with instead is to remove the current open/close features from the spec on the main branch, since it's pending a refactor in #470 anyways. We can then fix the negative literals the proper way, and then discuss if we want to compromise in order to enable the + and - syntax for spannables.

We have a significant refactor of the ABNF that's going to be needed due to all of the changes we agreed to in Seville and later.

We do need to incorporate negative number literals (which we have consensus on) at some point. I think your point, @stasm, is that this particular formulation is only needed if we use the sigil - in spannables.

I don't think this PR is necessarily the right friction point for addressing the sigil use. My suggestion would be (a) merge this (knowing that it's potentially throw-away); (b) figure out the more-radical syntax changes (specifically text-mode and namespacing) and implement those in the ABNF; (c) then address any changes for spannable (which might change the sigils). Isn't #470 sufficient as a friction point for the last?

Held for discussion in 2023-11-13 call

I believe this is now obsolete and should be closed in favor of #553

Yeah, this is out of date.