Example events should be tested against the schema, and failures rejected in CI.
Note that stubbed examples generated by scripts/eventlogging_legacy_schema_convert.js will then cause new schemas to fail until the example is filled in, or removed. I'm not sure whether that's desirable?
The only problem is that in schemas/event/secondary repository, jsonschema-tools tests is configured to skip the 'robustness' test for schemas in analytics/legacy, which include examples validation.
I'd love it if we could make jsonschema-tools tests more configurable, like a linter with the ability to enable/disable certain checks with more granularity, but that is a lot of work.
Thanks for the explanation, this makes sense!
Is there such a thing as potentially promoting some schemas out of "legacy" one day? From a quick search it looks like extensions.json:attributes.EventLogging.Schemas only points to "analytics/legacy" schemas, but is this a requirement when using the old EL clients? Would migrating away from "legacy" create a good opportunity to make breaking changes in a schema, clean up field names, remove some deprecated fields, etc.?
There's no promotion process; you'd just make a new schema and new instrumentation that calls mw.eventLog.submit (or whatever interface Metrics Platform cc @jlinehan advises you to) instead of mw.eventLog.logEvent. Basically this: https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To
But yes, when you do that it is a brand new instrumentation.