@colewhite, in https://phabricator.wikimedia.org/T288851#7456931 you said:

topics prefixed by rsyslog- will be automatically picked up by Logstash.

We've found using topic naming conventions for ingestion jobs to be brittle. We're moving towards using EventStreamConfig to automate configuring things like this. See: https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#consumers_and_producers

Example:

curl  'https://meta.wikimedia.org/w/api.php?action=streamconfigs&all_settings=1&streams=mediawiki.api-request' |  jq .

{
  "streams": {
    "mediawiki.api-request": {
      "topics": [
        "eqiad.mediawiki.api-request",
        "codfw.mediawiki.api-request"
      ],
      "stream": "mediawiki.api-request",
      "consumers": {
        "analytics_hadoop_ingestion": {
          "enabled": true,
          "job_name": "event_default"
        }
      },
      "canary_events_enabled": true,
      "topic_prefixes": [
        "eqiad.",
        "codfw."
      ],
      "destination_event_service": "eventgate-analytics",
      "schema_title": "mediawiki/api/request"
    }
  }
}

Here, we are declaring a consumer called 'analytics_hadoop_ingestion'. The settings for that consumer are arbitrary and specific to the consumer job. When that job runs, it requests all streams that have consumers.analytics_hadoop_ingestion declared, and uses those settings to import the data.

Logstash ingestion could probably do something similar, if the logging streams to import were declared in EventStreamConfig.

This would have been useful to debug T374662, aggregating the times out of elasticsearch is a bit hard as it would have to aggregate 50M requests through a single core (estimated time: multiple days due to repeated work for each pagination). Being able to throw hadoop at the problem would solve in a few 10s of minutes with an easy query.

Similar but different: T304373: Also intake Network Error Logging events into the Analytics Data Lake

This would be very useful for us to be able to understand if known problematic reusers (see: https://phabricator.wikimedia.org/T317001) are similarly saturating other data streams to avoid correcting issue behavior.

This would also help with some analysis in {T375146}

I've been told that this project would let me process Logstash data with SQL queries, and I would like that very much.