I'm not sure about the language here. Perhaps:

This seems to presume that "formatting" is a process that always produces a string. I don't think that's an accurate portrayal for this spec, which is supporting non-string formatting targets even if it does not explicitly define them.

No, that's a jump. It doesn't say anything about string resolution--it just says that the markup can be omitted ("unless otherwise provisioned"). It doesn't say what form any of that takes. We could expand on the examples, I suppose.

It's still referring to markup as "non-formattable", and states that "markup is omitted [unless] otherwise provisioned". Our design doc, on the other hand, includes this:

So they are formattable, and are not omitted by default except when formatting to a string.

Also, we did end up includes standalone, for which placeholders like {#img/} or {#button/} are rather likely to end up as concrete parts of the output

You're right, but then... how are img or button different from b?

I think what we might be trying to say is:

You're right, but then... how are img or button different from b?

They don't have a span of parts within the message that they're wrapping. They explicitly enable the representation of inline elements that should be called out in the explanation of markup.

Markup is an implementation-specific (but otherwise unsupported) mechanism [...]

What does this mean, and why does it need to be called out as the first thing about markup? How is markup "unsupported"?

My understanding of the MF2 position on markup is that we support syntax for it, and define how to represent it in non-string output. I don't really see how it could be more "supported" than that? Defining the output more precisely would've been much easier if we could have included formatted parts in the spec, but alas, that was not to be.

Maybe @mihnita already has some idea how the ICU4J implementation will handle markup, to give us another point of view?

I wrote the above hurriedly yesterday, regretting "unsupported" even as I suggested it. I don't think it's necessarily "wrong" though.

The problem with markup is that we don't provide real support for it. It doesn't use functions per-se and we don't provide anything in the registry for it. Open/close don't mean anything, other than as semantic constructs--we don't require them to be in order or paired.

What we're doing is reserving syntax for a partially specified feature. I think there are three ways that implementations will respond to this: (i) they won't support markup; (ii) they'll provide "pass through" support for it using the data model and/or some other means; or (iii) they'll implement either specific markup support (such as for HTML) or some form of pluggable (generic) support.

All implementations can check the syntax of a placeholder and can resolve any variables. Therefore all implementations MUST emit a syntax error for a malformed placeholder and MUST emit a resolution error if they encounter one.

Let's consider an implementation that supports markup. I think we should provide support for implementers who want to put additional requirements on (specific) markup, such as requiring open/close pairing, requiring open/close to be in order, require specific options (or validate option values). Users will appreciate it if a typo in their markup produces an appropriate error (@stasm has cited this in the past). This suggests that we provide an optional MarkupError that implementations MAY emit.

What about pass through? I think this is what we currently describe, albeit we require formatToString to remove the markup (pass through is to the data model or parts. Here we might want to relax the requirement to allow implementations to decide how to pass markup through (callers might need to reparse the output as a result).

What about non-supporting implementations? They might still emit data model parts or they might eat them. I think we might want to require that they not fail on markup but permit them to not emit the markup.

A key consideration is not to allow markup to become a security attack vector. We don't want <scr{#bad}ipt> to become a script tag (or other abusive sequence) in ways that are hard to detect. This can happen if the markup doesn't collapse to nothing until late.

The last two considered together might look like:

• An implementation SHOULD NOT emit the markup placeholder when formatting to string.
• An implementation MAY (SHOULD?) emit unrecognized markup placeholders in the data model (i.e. they do something other than formatToString)
• An implementation SHOULD NOT apply any additional validation (beyond syntax and resolution) to unrecognized or unsupported markup.
• An implementation MAY emit a MarkupError for unrecognized or unsupported markup. (this has to be allowed if implementations that support markup are to identify typos)

I feel strongly about this thread, but think we need to merge the overall changes into the spec. I am filing a separate issue to track this conversation.

I think we should make markup resolution a separate bullet in this list. Resolving markup is a very different operations from resolving expressions due to markup's not calling any functions, due to the default ignorable property in formatToString, and due to the fact that it can only happen inside patterns.

In fact, I'm not even sure if we should call it resolution. I guess we still resolve options, but crucially, in its current design, markup is encoded in the data model rather directly.

I think we should make markup resolution a separate bullet in this list. Resolving markup is a very different operations from resolving expressions due to markup's not calling any functions, due to the default ignorable property in formatToString, and due to the fact that it can only happen inside patterns.

I would prefer to keep these together, because the similarities are much greater than the differences.

• Markup resolution does not call external functions, but its results may be defined in the same way as for expressions. We also have expressions such as literals and variables which do not necessarily call any functions.
• Markup resolution uses the same option resolution as functions.
• The behaviour of markup during string formatting is a concern during formatting, but not resolution.
• Markup resolution only being used for pattern placeholders does not change how it is done.

In fact, I'm not even sure if we should call it resolution. I guess we still resolve options, but crucially, in its current design, markup is encoded in the data model rather directly.

Could you clarify what you mean by "data model" here? The only data model we define in detail is for the message before formatting, from which we need to select, resolve, and format a pattern during formatting. We left out formatted parts, so the output structure is not well defined.

My first reaction to the draft was like @stasm's: it's okay to be somewhat repetitious in a spec. Keeping a wall between the two similar-but-unlike concepts might be useful.

I didn't make that comment in the end, because there is a lot of commonality and the resolution into different buckets does happen later.

We also have expressions such as literals and variables which do not necessarily call any functions.

Are you sure? If there is a placeholder {|zebra|}, I'm pretty sure that invokes the :string formatting function. If there is a placeholder {$foo}, I'd allow the implementation to introspect foo's type, but I would expect some function to be called in the end. We don't want it to be an error to write messages like Hello {$user}...

If there is a placeholder {|zebra|}, I'm pretty sure that invokes the :string formatting function. If there is a placeholder {$foo}, I'd allow the implementation to introspect foo's type, but I would expect some function to be called in the end.

I agree that this is a reasonable thing for an implementation to do, but I don't think we should expect it to always happen. There are also reasonable further questions, such as: What formatter is called for {|zebra|} if an implementation allows for overriding default formatters, and :string is so overridden? Is it the core :string formatter, or the user-provided :string formatter that handles the zebra?

I think that's a question we ought to not resolve in this spec, leaving it instead to implementations.

I think that's a question we ought to not resolve in this spec, leaving it instead to implementations.

I agree. Hmm... what concrete change are we proposing with this thread?

Please move *(s option) back to the annotation production. It's helpful that variable and function define names. In fact, we should consider renaming them to veriable-name and function-name, respectively.

On the other hand, isn't it also useful for a function to have options? Previously, they were only a part of the annotation, which could effectively have any shape as private-use and reserved annotations.

I realize this is cosmetic (or academic?), but I think I was assuming we'd instead do:

pattern = *(text-char / text-escape / placeholder / markup)
placeholder = expression

That is, that markup won't end up being placeholders, but rather a new type of pattern elements, similar to how you implemented it in the data model. I presume that ABNF will have the most impact on shaping MF2's vocabulary, which is why I think we should be careful here.

Thoughts?

This part of the ABNF was explicitly included in the design doc, and so I'd really rather not change that here. I also think that having a catch-all name for "stuff in a pattern that has curly braces" is useful, and that "placeholder" works for that.

I would strongly prefer discussing this change as a separate follow-up to this PR.

This works, but aesthetically I'd prefer keeping the shallower approach and just duplicating "options": { "type": "array", "items": { "$ref": "#/$defs/option" } } (it's nice to have an easily-referenced "option" type).

Could you clarify what use case you have in mind for an externally-referrable JSON schema option type?

In my experience (and according to the reference docs), these $defs are meant only for internal use within this schema, and that only the top-level entity here should be considered as an external API.

As I said, it's just aesthetic. Referencing a singular is both shallower and a bit easier to keep in my human memory when contemplating the schema.

This doesn't look quite right for a use of RFC 2119 "MUST".

The usage here is intentional, at least on my part: An implementation must not allow for the resolution of any {#foo} part to ever fail.

what order is this list in? Probably we should sort the reserved sigils into code point order?

I think it's currently something close to the layout order on a US keyboard? I've no opinion on the order here.

What I'm saying is: it's not "afterwards" at all, it might be now as part of the resolution, if the implementation chooses to make it work that way.

Instead of saying this (which doesn't really help implementations either way), perhaps state what our limits are:

Yes.

As noted above, we should probably sort these

missing markup-standalone ?

It's there as ["/"] in the first line. It's not a separate production because it would require an infinite lookahead.