"Where appropriate" ... "MUST" seems difficult to implement/test.

I think we should say when it is appropriate. I believe this might be:

When a message part has a direction different from the message pattern string or when the message part's direction is not known...

I agree that "Where appropriate ... " is far too vague to be a conformance MUST.

However, condition is too strong.

1. There are many instances where isolation is not required, such as a single Arabic word in a placeholder surrounded by Latin. It would be better to supply a functional description. Off the top of my head: w
   • Where the text of the placeholder or surrounding text would rearrange due to the insertion.
2. There is a significant number of products in use that do not yet support isolates. [Aside: I really wish I'd made the embeddings be isolating in the first place, but water far under the bridge.] For use of MF2.0 with those products, we have to have a fallback position (ugly as that is).

I've updated the text following @aphillips's suggestions.

@macchiati Is there an existing algorithm for determining when isolation would not be required? I presume that this would require scanning both the placeholder's stringified value as well as that of its surroundings, and ensuring that neither have any weakly typed characters immediately adjacent to the other.

I would prefer an algorithm that would not require such introspection, even if it does come at a cost of needing to include isolating characters in some situations where they have no effect.

To support products and situations which do not support isolates, would it be sufficient to mandate the default behaviour, and allow for a formatter to provide alternatives? For example, in the JS implementation, I could imagine including an option like:

type MesageFormatOptions = {
  isolate = 'isolate' | 'embed' | 'none' = 'isolate',
  ...
}

This would allow for defaulting to isolation (explicitly spec-defined), while also being able to provide implementation-defined fallback solutions.

I think we are thinking along similar lines.

For any hard conformance requirement, I think we should be precise as to the minimum required. That is relatively simple to apply, as long as an implementation has access to an api for the bidi algorithm. And someone could do an analysis to try to come up with a simpler algorithm for this specific task (but such an algorithm would take a lot of testing — there are many edge cases).

However, what we can do make it clear that an implementation can "over-isolate"; add codes that are not needed — the most extreme case would be to isolate every placeholder at every level. That would be an approach that anyone could implement. And there could be intermediate approaches.

"the formatted string representation" doesn't say what happens with formatToParts. Do message segments have lang/dir metadata or do they include isolating controls? I'd prefer metadata (so that one could implement markup instead of controls)

+1 for documenting the parts behavior.
I also prefer metadata, or special parts.
Whatever the "format to parts" will produce, but not raw Unicode bidi controls.

For the 'formatToString' functionality, I think we have no choice but to deploy the bidi controls. For the full data model, I agree that metadata is better. (One can argue that the formatToString is just semantic sugar, but I think the end result of processing for formatToString() had better agree with that.)

I've left out the formatted parts representation from this PR because I think that's a separate concern, and a topic on which I find us to have less agreement. In fact, I would be (pleasantly!) surprised if we were able to define a common formatted-part definition that could be shared by all implementations, given how much the design of such an implementation is tied to its surroundings.

Rather than discussing this further here, we should continue in #41 or #272.

I think there was agreement that the format to parts would return the bidi info as some kind of part / meta on parts, rather than bidi control characters.
If we have agreement on that, can we say it in the spec?
Or did I misunderstand it and there is no agreement?

I agree that "Where appropriate ... " is far too vague to be a conformance MUST.

However, condition is too strong.

1. There are many instances where isolation is not required, such as a single Arabic word in a placeholder surrounded by Latin. It would be better to supply a functional description. Off the top of my head: w
   • Where the text of the placeholder or surrounding text would rearrange due to the insertion.
2. There is a significant number of products in use that do not yet support isolates. [Aside: I really wish I'd made the embeddings be isolating in the first place, but water far under the bridge.] For use of MF2.0 with those products, we have to have a fallback position (ugly as that is).

For the 'formatToString' functionality, I think we have no choice but to deploy the bidi controls. For the full data model, I agree that metadata is better. (One can argue that the formatToString is just semantic sugar, but I think the end result of processing for formatToString() had better agree with that.)

@macchiati With reference to your comment #315 (comment), would this be the sort of change you're looking for? Or have I perhaps understood your "minimum" incorrectly?

The intent here would be to reduce the complexity required of implementations, such that they only need to add isolates if they explicitly know that a message and a part have different directionality. For instance, if an instance doesn't have the capability to determine the directionality of a variable $foo, it would not be required to isolate it.

To be clear, my own preference would be to require isolation by default in that case, but to allow for an implementation to be configurable to act differently, isolating either less, more, or somehow differently than defined in the spec.

such that they only need to add isolates if they explicitly know that a message and a part have different directionality. For instance, if an instance doesn't have the capability to determine the directionality of a variable $foo, it would not be required to isolate it.

This is fundamentally the problem. In order to know that a given part has a different direction and/or requires isolation, you have to run the bidi algorithm over the assembled string and see if it generates multiple "runs" and that run boundaries fall on placeholder boundaries.

To be clear, my own preference would be to require isolation by default in that case, but to allow for an implementation to be configurable to act differently, isolating either less, more, or somehow differently than defined in the spec.

+1

The spec could be written to isolate by default, but allow implementations to optimize their output (that is, omit isolation if they determine it is safe to do so)

I suspect this means that the MUST on line 26 should either be amended with the out I just mentioned or become SHOULD... unless...

I think the best way to state the requirement in L24 and beyond is something like the following.

The formatted string representation MUST ensure that when the BIDI algorithm is applied to the message, the part must behave as if it were surrounded by the appropriate isolate characters. The prefixed isolate character is:

• U+2066 LEFT-TO-RIGHT ISOLATE if the part is known to have LTR directionality,
• U+2067 RIGHT-TO-LEFT ISOLATE if the part is known to have RTL directionality, or
• U+2068 FIRST STRONG ISOLATE if the part's directionality is not certain.

The postfix isolate character is U+2069 POP DIRECTIONAL ISOLATE.

NOTE: This requirement does not imply that BIDI isolates must be present in the resulting text:

• In the vast majority of cases, they are not necessary and can be omitted. Implementations can have more or less sophisticated algorithms to determine this, but they need to default to using the isolates (or equivalent mechanisms) wherever there is any question.
• Moreover, many implementations support the older BIDI embedding characters and BIDI marks, so an implementation may need to use a combination of those instead.
  The important requirement is the behavior, not the mechanism used to achieve it.

I would probably call this something more generic, such as "Handling Bidirectional Text"

I would reverse the normative statements for clarity to implementers:

Implementations must provide at least one isolation mechanism from the list below. They may provide additional mechanisms in order to allow better overall control from users.

At the moment, the PR only presents one mechanism/strategy. If we add the compatibility mode I mention in an earlier comment, I would prefer either requiring both or at least the compatibility mode of all implementations, rather than "at least one".

I also think that there's something not right in how this is current phrased. MAY provide one or more... and one MUST behave as follows... seem to be at odds with each other? Can an implementation provide zero strategies?

I liked the wording suggested by @macchiati in #315 (comment). @aphillips's suggestion above also sounds good to me.

This leaves a gap for same-base-paragraph-direction placeholders with various edge effects (generally starting or ending with weak runs or runs with neighboring character issues). I admit that we've wrapped ourselves up trying to define when we are required to isolate without having to run the bidi algorithm. I don't have a solution yet, so perhaps keeps this for now, but add a note about the issue??

I would split the term definition of "formatted part" from the requirement.

I don't think I see the hole you're describing. Could you give an example?

The pattern string is:

السعر {$price} + {$shipping_amt} الشحن

The currency code for both is USD and the symbol for that is USD in the ar-AE locale. The resulting string is:

السعر 1,234.56 AED + 12.99 USD الشحن

It should be:

السعر ⁧1,234.56 USD⁩ + ⁧12.99 USD⁩ الشحن

Here are the code point sequences with and without RLI/PDI around the prices:

without: \u0627\u0644\u0633\u0639\u0631 1,234.56 USD + 12.99 USD \u0627\u0644\u0634\u062d\u0646
with: \u0627\u0644\u0633\u0639\u0631 \u20671,234.56 USD\u2069 + \u206712.99 USD\u2069 \u0627\u0644\u0634\u062d\u0646

In both cases the base paragraph direction of the pattern string and also each of the placeables is RTL, but the appearance of weak (numbers) and strong (letters) LTR runs confuses bidi unless isolated.

Dates often have this sort of thing too or any string that starts or ends with numeric sequences or enclosing punctuation.

Ok, I see it now, how the formatted value might be nominally RTL but only composed of neutral and LTR characters. It's interesting to play around with this in different browsers, as I see this:

const usd = new Intl.NumberFormat('ar-AE', { style: 'currency', currency: 'USD'}).format(12.99)

// in Chrome & Node.js
usd === "US$ 12.99"
usd.split('').map(c=>c.charCodeAt(0))
// [ 85, 83, 36, 160, 49, 50, 46, 57,  57 ]

// in Firefox
usd === "‏12.99 US$"
usd.split('').map(c=>c.charCodeAt(0))
// [ 8207, 49, 50, 46, 57, 57, 160, 85, 83, 36 ]

Specifically, Firefox prefixes the string with an RTL mark for currencies. It doesn't really help with the isolation, though.

So I agree that this situation is left outside the "must" statement, I think it's potentially ok? As in, at least we don't have a "must not" so that an implementation is allowed to introspect the formatted part and determine that it ought to be isolated.

I would add one more callout here. Many of the formatted parts are produced by a formatter with a given locale. APIs exist in at least a few platforms for determining the default base direction based on locale. So perhaps:

An implementation must determine the base paragraph direction for each formatted part. These might include a metadata field/method, introspection of the formatted part using UBA [UAX9], or from the locale of the placeholder's formatter (for example, Intl.Locale's textInfo in JavaScript). Formatters are responsible for using whatever mechanism is provided by the implementation to provide appropriate directional information for each formatted part.

You're assuming that the message formatter's locale matches the language of the pattern, which is a bad assumption. Many (most!) resource systems fall back or allow sparse population of translated resources. As a result, the locale of the message formatter (and its subformatters) may not match that of the pattern string. This is a feature. It is often the case that developers will wish to test code in a given locale before the translations are available or to use pseudo.

I also note that this paragraph is not an example of the preceding requirement. Perhaps insert this just ahead of this paragraph:

Implementations should not isolate formatted parts when the base paragraph direction of the pattern and the formatted parts match and there are no opposite direction runs of text in the formatted part.

Let me give an example to verify if I've understood your concern right:

An app using English as its fallback locale is partially localized to Arabic, with some English messages filling in holes. If such a LTR message is formatted using an RTL formatter and it contains a placeholder with a variable reference to an RTL value, we should isolate the placeholder's value in the output, even though the formatter and the value are both RTL.

If this is the case, then I think what we need is a more explicit definition somewhere of the "message locale", which I would consider to be English in the above example, even if the formatter is nominally in Arabic.

That's more or less effectively what I'm saying. The challenge is that (let's use Java as an example), I have code that looks like this:

ResourceBundle myPatterns = ResourceBundle.getBundle("com.example.Patterns", myRuntimeLocale);
MessageFormat2 formatter = new MessageFormat2(myPatterns.getString("somePattern"), myRuntimeLocale);

If you query myPatterns for its locale, you get myRuntimeLocale, even though the bundle is really a stack of bundles. If myRuntimeLocale is ar-AE, the actual translations probably live in the ar bundle. If the developer has added a new string, though, somePattern may be coming from the ROOT locale bundle (and be in, e.g. English). Both formatter and its subformat patterns are trying to serve the ar-AE locale, though and finding out which specific locale bundle coughed up the pattern is both inaccessible to it (MF2 doesn't know that the pattern is coming from a bundle--it's just a string, as far as it is concerned) and doesn't add any value to instrument in most cases.

The actual language (as opposed to the asserted language/locale of it) is not really a concern for MF2, as long as the pattern is valid. If we wanted to do language-sensitive processing (capitalization, for example), we'd still use a locale for that.

When it comes to bidi, it also doesn't matter as much as the base paragraph direction does, save that the language might serve as a hint for what the direction should be (lacking other information).

I also think that there's something not right in how this is current phrased. MAY provide one or more... and one MUST behave as follows... seem to be at odds with each other? Can an implementation provide zero strategies?

I liked the wording suggested by @macchiati in #315 (comment). @aphillips's suggestion above also sounds good to me.

I suggest we not use part to mean the stringified result of formatting. We've sort of established formatToParts() as the name of the method which returns structured formatting results, which implies that a part is one such structured result of formatting. Perhaps formatted elements or formatted pattern elements would work better?