Change 498388 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498388

Change 498388 merged by Gehel:
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498388

disabling this logger for now, let's not forget to re-enable it once we've fixed the underlying issues!

Change 498395 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498395

Change 498395 merged by Gehel:
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498395

Mentioned in SAL (#wikimedia-operations) [2019-03-22T15:26:58Z] <gehel> restarting elasticsearch on elastic1046 for logging configuration change - T218994

The elasticsearch security manager is preventing log4j2 to auto-reload it's configuration (more precisely, it can't restart the GELF appender, as socket access is denied). So we will require a full cluster restart to reload the logging configuration. This will be done next week, bundled with the JVM upgrade.

Change 498420 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Rename _retry_on_conflict to retry_on_conflict

https://gerrit.wikimedia.org/r/498420

Can't we apply the setting directly to elasticsearch cluster settings?

We already have this in the cluster settings:

"logger" : {
  "org" : {
    "elasticsearch" : {
      "cluster" : {
        "routing" : {
          "allocation" : "INFO"
        }
      },
      "http" : "DEBUG",
      "transport" : "DEBUG"
    }
  },
  "io" : {
    "netty" : "DEBUG"
  },
  "transport" : "DEBUG"
}

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'

In T218994#5048792, @EBernhardson wrote:

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'

So simple. And yes, we can!

Mentioned in SAL (#wikimedia-operations) [2019-03-25T10:40:00Z] <gehel> disable deprecation warnings on elasticsearch eqiad - T218994

Mentioned in SAL (#wikimedia-operations) [2019-03-25T19:48:14Z] <dcausse> elasticsearch search cluster: SET "logger.org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder" to "ERROR" to chi/psi/omega@eqiad (T218994)

Mentioned in SAL (#wikimedia-operations) [2019-03-25T19:54:31Z] <dcausse> elasticsearch search cluster: SET "logger.org.elasticsearch.common.logging.DeprecationLogger" to "ERROR" to psi/omega@eqiad (T218994)

Went through logstash and filtered one error at a time to collect a list of new warnings. Task description has been updated will all of the warning messages, transient cluster settings have updated the following channels to ERROR level on all six cirrus clusters

• org.elasticsearch.deprecation.search.sort.GeoDistanceSortBuilder
• org.elasticsearch.deprecation.common.ParseField
• org.elasticsearch.common.logging.DeprecationLogger
• org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder
• org.elasticsearch.deprecation.index.similarity.SimilarityService

Once we have resolved the warnings in the task description these should all be reverted back to INFO.

Mentioned in SAL (#wikimedia-operations) [2019-03-25T20:32:29Z] <ebernhardson> T218994 set various deprecation channels on all six cirrus elasticsearch clusters to ERROR

Change 498380 merged by jenkins-bot:
[search/MjoLniR@master] [elastic] fix deprecation warning on retry_on_conflict

https://gerrit.wikimedia.org/r/498380

Most of the warnings have been muted in elastic logging, moving forward we need to unmute to start up the stream of deprecation errors again.

Cleared out out custom settings for deprecation logging on -eqiad and -codfw clusters, i haven't changed cloudelastic. Two high volume deprecations are still going (and were re-muted), will see if other low volume deprecations show up over time:

• T219265: Remove leading _ in _retry_on_conflict. Previously resolved but we haven't deployed the config to enable the fix. Will ship a config patch next week
• T209859: negative weights wont be allowed any more, previously captured in ticket.

I've also setup cirrus running against 6.8.18 locally, added tickets for deprecations triggered by the integration tests:

• T288764: Set include_type_name in all get requests
• T288765: Always provide minimum_should_match in bool queries.

Change 787068 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/787068

Change 787068 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/787068

Change 787074 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Forward CirrusSearchDeprecation logs to logstash

https://gerrit.wikimedia.org/r/787074

Change 787074 merged by jenkins-bot:

[operations/mediawiki-config@master] Forward CirrusSearchDeprecation logs to logstash

https://gerrit.wikimedia.org/r/787074

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:12:13Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787074|Forward CirrusSearchDeprecation logs to logstash (T218994)]] (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:13:33Z] <ebernhardson@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:787068|cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:16:17Z] <ebernhardson@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: Revert: [[gerrit:787068|cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 58s)

Change 788282 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps in deployment-prep

https://gerrit.wikimedia.org/r/788282

Change 788282 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps in deployment-prep

https://gerrit.wikimedia.org/r/788282

Change 790408 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/790408

Change 790408 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/790408

Mentioned in SAL (#wikimedia-operations) [2022-05-09T20:36:25Z] <cjming@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:790408|cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 51s)

Reviewing production logs for the week since we turned on deprecation logging, there are very few remaining:

• Should clauses in the filter context will no longer automatically set the minimum should match to 1 in the next major version.. We missed something when doing T288765.
  • Comes from api requests issued by MediaSearch
  • Comes from CirrusSearch web requests as well, potentially reproduced when combining text + insource keyword. Might also require invoking the PhraseSuggestFallbackMethod: https://en.wikipedia.org/w/index.php?sort=random&search=foobar+insource%3Aqwerty&title=Special%3ASearch&fulltext=1&ns0=1&cirrusDumpQuery
  • Might be other sources, not entirely clear. The above two make up the majority of cases at least.
• ApiFeatureUsage triggers "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.". When we undeploy ApiFeatureUsage these queries will stop being issued.

Change 793124 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] metastore: Update deprecated version parameters

https://gerrit.wikimedia.org/r/793124

Change 793124 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] metastore: Update deprecated version parameters

https://gerrit.wikimedia.org/r/793124

New warning in logstash I hadn't seen before, will need to look into how metastore is using versioning but i suspect this should be minor.

warning: 299 Elasticsearch-6.8.23-4f67856 "Usage of internal versioning for optimistic concurrency control is deprecated and will be removed. Please use the `if_seq_no` and `if_primary_term` parameters instead. (request for index [mw_cirrus_metastore], type [_doc])"

Related stack frames:

#6 /srv/mediawiki/php-1.39.0-wmf.13/vendor/ruflin/elastica/lib/Elastica/Index.php(194): Elastica\Client->addDocuments(array, array)
#7 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/includes/MetaStore/MetaSaneitizeJobStore.php(90): Elastica\Index->addDocuments(array)
#8 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(348): CirrusSearch\MetaStore\MetaSaneitizeJobStore->update(Elastica\Document)
#9 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(251): CirrusSearch\Maintenance\SaneitizeJobs->updateJob(Elastica\Document)
#10 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(89): CirrusSearch\Maintenance\SaneitizeJobs->pushJobs()
#11 /srv/mediawiki/php-1.39.0-wmf.13/maintenance/includes/MaintenanceRunner.php(119): CirrusSearch\Maintenance\SaneitizeJobs->execute()
#12 /srv/mediawiki/php-1.39.0-wmf.13/maintenance/doMaintenance.php(84): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(419): require_once(string)

Change 802654 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] metastore: Don't use versioning semantics for saneitize

https://gerrit.wikimedia.org/r/802654

Change 802654 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] metastore: Don't use versioning semantics for saneitize

https://gerrit.wikimedia.org/r/802654

Checked logstash with the query channel:CirrusSearchDeprecation AND NOT exception.trace:"ApiFeatureUsage" and verified nothing is coming through

We had a few custom log levels set in the elasticsearch cluster state, specifically logger.org.elasticsearch.deprecation.common.ParseField and logger.org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder were muted. I've unmuted them and nothing seems to be complaining (and they used to be quite verbose). There is reindexing currently going on so if the reindexing process triggers any deprecations those should come up today. If nothing comes up this can be considered complete.

Looks like wikidata is still triggering the should clause warning (rarely):

https://www.wikidata.org/w/api.php?format=xml&prop=info%7Cpageprops&action=query&generator=search&ppprop=wikibase_item&gsrsearch=e&gsrsort=random

warning: 299 Elasticsearch-6.8.23-4f67856 "Should clauses in the filter context will no longer automatically set the minimum should match to 1 in the next major version. You should group them in a [filter] clause or explicitly set [minimum_should_match] to 1 to restore this behavior in the next major version."

Adjusting the query a bit, this only triggers the warning when using random sort mode (which makes sense, it puts the whole query into filter context).

Change 816849 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/WikibaseCirrusSearch@master] Apply minimum_should_match for elastic 7

https://gerrit.wikimedia.org/r/816849

Change 816849 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@master] Apply minimum_should_match for elastic 7

https://gerrit.wikimedia.org/r/816849

New batch of logs came out Aug 4th, but they don't really make much sense. I would expect this error message to come from a mapping update, but the stack trace clearly shows this was run while sending writes. As this is the ElasticaWrite job it's highly likely this is the cloudelastic cluster, but it could be one of the normal clusters. Will write up a quick script to iterate over all indices in all clusters and check if anything still has an all field defined.

warning: 299 Elasticsearch-6.8.23-4f67856 "[_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field."

from /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Elastica/DeprecationLoggedHttp.php(32)
#0 [internal function]: CirrusSearch\Elastica\DeprecationLoggedHttp->CirrusSearch\Elastica\{closure}(resource, string)
#1 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php(166): curl_exec(resource)
#2 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Request.php(194): Elastica\Transport\Http->exec(Elastica\Request, array)
#3 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Client.php(689): Elastica\Request->send()
#4 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Bulk.php(358): Elastica\Client->request(string, string, string, array, string)
#5 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/DataSender.php(322): Elastica\Bulk->send()
#6 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(160): CirrusSearch\DataSender->sendData(string, array)
#7 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/JobTraits.php(138): CirrusSearch\Job\ElasticaWrite->doJob()
#8 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Updater.php(446): CirrusSearch\Job\CirrusGenericJob->run()
#9 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Updater.php(221): CirrusSearch\Updater->pushElasticaWriteJobs(array, Closure)
#10 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Updater.php(79): CirrusSearch\Updater->updatePages(array, integer)
#11 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/LinksUpdate.php(51): CirrusSearch\Updater->updateFromTitle(Title)
#12 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/JobTraits.php(138): CirrusSearch\Job\LinksUpdate->doJob()
#13 /srv/mediawiki/php-1.39.0-wmf.22/extensions/EventBus/includes/JobExecutor.php(79): CirrusSearch\Job\CirrusTitleJob->run()
#14 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#15 {main}

I wrote the small python script, P32409, to check for _all fields across the cluster. The only place that has _all fields are the historical apifeatureusage indices. I'm not sure how the above stack trace came about, but it doesn't seem like something we need to worry about.