why is nmtoken more relaxed about starters than name? Is it because we want to allow digits??

Note that nmtoken is basically an unquoted literal which we provide for convenience, e.g. so you can say:

when foo {{$count :number option=bar}}

instead of having to quote foo and bar as (foo) and (bar)

I guess what I'm digging at here is whether we can simplify parsing rules given that we're mostly breaking on whitespace or on specific characters like =. The less we use specific character lists or rules, the easier it is to implement.

name and nmtoken are taken directly from https://www.w3.org/TR/xml/#NT-Name. And yes, the goal was to make it convenient to use (a) numerical option values (fractionDigits=1) and (b) option values such as year=2-digit.

More broadly, the use of nmtoken is intended to align MF2 with LDML. CLDR data is defined as attribute enums (e.g. in ldmlSupplemental.dtd), which implies that they are Nmtokens.

nmtoken however is actually exposed to more than just LDML/CLDR constructs. Developers of selectors and formatters will use them to define key values and option values. That is, our user base is wider than just CLDR stuff or ICU developers. If we define values that are a strict superset of those restrictions then any XML, LDML, or CLDR values will work. LDML being a dialect of XML means that they are restricted by XML, but we don't have to be. Thoughts?

Wait, are you suggesting that keys and option values should be even less restricted than nmtoken?

My goal here (last year when I wrote this for my proposal) was to make sure that whatever appears in LDML attlists (now or in the future) can be a valid key or an option value. That's why I chose Nmtoken for these two productions.

Yes. I'm suggesting that any characters we omit we know why we are omitting them. Some implementations will probably not check the values so long as they are able to parse the various tokens (which don't really depend on the characters inside the token); others will pedantically check the character ranges.

To be honest, most developers will write enumerated values in ASCII for keys and option values. But for cases when they don't the restrictions on users should make sense in our world (and not just because someone else did it)

@aphillips

LDML being a dialect of XML

I hope you meant "LDML being an application of XML". XML doesn't have any dialects (except for XML 1.1, maybe).

@duerst exactly so

Similar to my comment below, I think these rules would be more readable with conventional OWS and RWS (as in "{optional,required} white space") rules, and column-aligned as in RFC 5234.

and so on.

Can we discuss using RWS and OWS in a separate PR? They touch every production in the ABNF.

I'm against column alignment for as long as we expect the grammar to change. They generate needless diffs. Let's do it once when the grammar stabilizes.

What are the expected semantics of an option value that is a nmtoken but not a name, as in e.g. {:func foo=1}?

It parses as a literal, "1". The implementation of :func can interpret it as a number if it makes sense to do so for the foo option.

Hrm, so the value of an option is either a variable or a literal, but the literal can be implicit rather than quoted? Are {:func foo=|1|} and {:func foo=1} therefore indistinguishable?

There's a super-subtle difference between nmtoken and literal values.

An nmtoken value might be validated at parse time and the values that can be present in an nmtoken are restricted vs. the values permitted in a literal. The use of numbers is fairly common in existing formatters, Cf. Intl.NumberFormat options such as maximumSignificantDigits. But other values have limited (and enumerated) values which might be validated at parse time.

A literal value probably is a parsing error (invalid argument) when the function wants a number or enumerated value. MF's options are untyped, but the underlying implementation might not be.

There may be a "tripping hazard" here for users who can't see the difference between:

{:func symbol=US$} (invalid, as $ is reserved) and {:func symbol=|US$|}

Hrm, so the value of an option is either a variable or a literal, but the literal can be implicit rather than quoted?

Yes.

Are {:func foo=|1|} and {:func foo=1} therefore indistinguishable?

As far as I recall we have not had an explicit discussion on this, but my position would be that during formatting the :func handler should not be able to distinguish these two from each other.

Thanks for suggesting this. As I said above, I'd prefer to not column-align the ABNF for now because I don't think the names of production are final and I expect the next few weeks to bring a few changes.