Change #1058574 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: staging config for modernized rec-api

https://gerrit.wikimedia.org/r/1058574

Change #1058574 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: staging config for modernized rec-api

https://gerrit.wikimedia.org/r/1058574

Change #1059071 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[operations/deployment-charts@master] ml-services: override staging rec-api entrypoint

https://gerrit.wikimedia.org/r/1059071

Change #1059071 merged by Ilias Sarantopoulos:

[operations/deployment-charts@master] ml-services: override staging rec-api entrypoint

https://gerrit.wikimedia.org/r/1059071

The modernized recommendation API has been deployed on LiftWing staging. It is currently available through an internal endpoint that can only be accessed by tools that run within the WMF infrastructure:

kevinbazira@deploy1003:~$ time curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[{"title":"Applecrab","pageviews":0,"wikidata_id":"Q19595924","rank":6.0,"langlinks_count":0},{"title":"Flamenco (apple)","pageviews":0,"wikidata_id":"Q19597233","rank":20.0,"langlinks_count":1},{"title":"Malus × zumi","pageviews":0,"wikidata_id":"Q5990804","rank":28.0,"langlinks_count":3}]
real	0m0.443s
user	0m0.004s
sys	0m0.009s

@santhosh, please test it and let us know of any edge cases you may come across. Once you have confirmed that there are none, we shall prepare to move it to production and provide an external endpoint.

Change #1059549 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/extensions/ContentTranslation@master] refactor: Add source and target language parameters to API request

https://gerrit.wikimedia.org/r/1059549

@kevinbazira This looks great. There is a minor blocker for production deployment though. Our CX client side code sends query params as s,t, n etc. And new API does not accept them. I submitted a patch for this and if it goes with this week's train, we should be able to deploy new API by early next week.

There is another way to do this, by adding support for legacy param names to recommendation api, which I don't like. As CX is the only consumer for this tool, I think it is better to move forward and not support legacy param names.

cc @KartikMistry

Change #1059549 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] recommendation api: migrate to new API param names

https://gerrit.wikimedia.org/r/1059549

Thank you for the update @santhosh. As we were testing the modernized recommendation-api endpoints to make sure there is parity between the LiftWing staging instance and the CloudVPS instance, we noticed that the section recommendation endpoint returns a 505 error as shown below:

kevinbazira@deploy1003:~$ time curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/v1/translation/sections?source=en&target=es&seed=Apple&search_algorithm=morelike&count=3"
Internal Server Error
real	0m0.882s
user	0m0.005s
sys	0m0.009s

This is likely caused by get_section_suggestions() not setting a Host header. Following our discussion in 1052445#e041c349_eeda96d0, since there is use of CXSERVER_URL, adding CXSERVER_URL_HEADER in the config and using it to fetch section suggestions will enable this endpoint to run on LiftWing.

Change #1060192 had a related patch set uploaded (by Santhosh; author: Santhosh):

[research/recommendation-api@master] Add CXSERVER_HEADER config

https://gerrit.wikimedia.org/r/1060192

@kevinbazira I added CXSERVER_HEADER config value to match the env values in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1058574/2/helmfile.d/ml-services/recommendation-api-ng/values-ml-staging-codfw.yaml#23

Change #1060192 merged by jenkins-bot:

[research/recommendation-api@master] Add CXSERVER_HEADER config

https://gerrit.wikimedia.org/r/1060192

Change #1060377 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: use cxserver host header in rec-api

https://gerrit.wikimedia.org/r/1060377

Change #1060377 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: use cxserver host header in rec-api

https://gerrit.wikimedia.org/r/1060377

@santhosh, thank you for adding the cxserver host header config. We deployed the new image and tested the section recommendation endpoint but still got a 505 error:

kevinbazira@stat1008:~$ curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/v1/translation/sections?source=en&target=es&seed=Apple&search_algorithm=morelike&count=3"
Internal Server Error
real	0m0.894s
user	0m0.047s
sys	0m0.004s

@klausman helped troubleshoot this issue on LiftWing staging and we found that an extra slash is added when creating section_suggestion_api. Surprisingly the URL with an extra slash works well with the cxserver external endpoint but doesn't work well with the cxserver k8s internal endpoint as shown below:

# external endpoint works with extra slash
kevinbazira@deploy1003:~$ curl https://cxserver.wikimedia.org//v2/suggest/sections/Apple/en/fr
{"sections":{"sourceLanguage":"en","sourceTitle":"Apple","targetLanguage":"fr","targetTitle":"Pomme","sourceSections":["Etymology","Description","Taxonomy","Cultivation","Toxicity","Uses","In culture","See also","References","Further reading","External links"],"targetSections":["Étymologie","Botanique","Typologie","Histoire","Économie","Utilisation","Symbolique","Musées et foires","Notes et références","Voir aussi"],"present":{"Etymology":"Étymologie","Uses":"Utilisation","See also":"Voir aussi","References":"Notes et références"},"missing":{"Description":"Conception","Taxonomy":"Taxonomie","Cultivation":"Culture","Toxicity":"Toxicité","In culture":"Dans la culture","Further reading":"Bibliographie","External links":"Liens externes"}}}
kevinbazira@deploy1003:~$ 
kevinbazira@deploy1003:~$ 

# internal endpoint doesn't work with extra slash
kevinbazira@deploy1003:~$ curl http://localhost:6015//v2/suggest/sections/Apple/en/fr
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET //v2/suggest/sections/Apple/en/fr</pre>
</body>
</html>
kevinbazira@deploy1003:~$ 
kevinbazira@deploy1003:~$ 

# internal endpoint works without extra slash
kevinbazira@deploy1003:~$ curl http://localhost:6015/v2/suggest/sections/Apple/en/fr
{"sections":{"sourceLanguage":"en","sourceTitle":"Apple","targetLanguage":"fr","targetTitle":"Pomme","sourceSections":["Etymology","Description","Taxonomy","Cultivation","Toxicity","Uses","In culture","See also","References","Further reading","External links"],"targetSections":["Étymologie","Botanique","Typologie","Histoire","Économie","Utilisation","Symbolique","Musées et foires","Notes et références","Voir aussi"],"present":{"Etymology":"Étymologie","Uses":"Utilisation","See also":"Voir aussi","References":"Notes et références"},"missing":{"Description":"Conception","Taxonomy":"Taxonomie","Cultivation":"Culture","Toxicity":"Toxicité","In culture":"Dans la culture","Further reading":"Bibliographie","External links":"Liens externes"}}}
kevinbazira@deploy1003:~$ 
kevinbazira@deploy1003:~$

When we changed f"{configuration.CXSERVER_URL}/v2/suggest/sections/" to f"{configuration.CXSERVER_URL}v2/suggest/sections/" in the recommendation/external_data/fetcher.py file on the staging container, fetching section suggestions worked as shown below:

$ curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/v1/translation/sections?source=en&target=es&seed=Apple&search_algorithm=morelike&count=3"
[{"source_title":"Malus","target_title":"Malus","source_sections":["Description","Subdivisions and species","Cultivation","Toxicity","Uses","Gallery","References","External links"],"target_sections":["Características","Taxonomía","Referencias","Bibliografía","Enlaces externos"],"present":{"References":"Referencias","External links":"Enlaces externos"},"missing":{"Description":"Descripción","Cultivation":"Cultivo","Uses":"Usos","Gallery":"Galería","Subdivisions and species":"Subdivisions and species","Toxicity":"Toxicity"}},{"source_title":"Pear","target_title":"Pyrus","source_sections":["Etymology","Description","History","Major species","Cultivation","Production","Storage","Uses","Nutrition","Cultural references","Gallery","See also","References","Further reading","External links"],"target_sections":["Etimología","Historia","Descripción","Distribución y hábitat","Taxonomía","Selección de especies","Usos","Referencias","BIbliografía","Enlaces externos"],"present":{"Etymology":"Etimología","Description":"Descripción","History":"Historia","Uses":"Usos","References":"Referencias","External links":"Enlaces externos"},"missing":{"Cultivation":"Cultivo","Production":"Producción","Nutrition":"Nutrición","Cultural references":"En la cultura popular","Gallery":"Galería","See also":"Véase también","Further reading":"Bibliografía","Major species":"Major species","Storage":"Storage"}},{"source_title":"Malus sieversii","target_title":"Malus sieversii","source_sections":["Characteristics and growth","History and importance","Renewed interest","Conservation","Gallery","References","External links"],"target_sections":["Descripción","Taxonomía","Referencias","Enlaces externos"],"present":{"References":"Referencias","External links":"Enlaces externos"},"missing":{"Conservation":"Conservación","Gallery":"Galería","Characteristics and growth":"Characteristics and growth","History and importance":"History and importance","Renewed interest":"Renewed interest"}}]
kevinbazira@deploy1003:~$ 
kevinbazira@deploy1003:~$ 

Please add this fix to: https://github.com/wikimedia/research-recommendation-api/blob/ef1288fee3ffcf5266f095184fea7b048bb6e1e2/recommendation/external_data/fetcher.py#L293C30-L293C82

One thing that occurred to me: in filesystem-land, many languages/stdlibs have a path joining function (in Python it's os.path.join()). One of the things beyond protability (/ vs \ as path separators) is normalization, i.e. reducing runs of / to single slashes. Maybe Python or the framework(s) used has a similar function for URLs?

Change #1061662 had a related patch set uploaded (by Santhosh; author: Santhosh):

[research/recommendation-api@master] Avoid duplicate slash in URLs for cxserver API

https://gerrit.wikimedia.org/r/1061662

Change #1061662 merged by jenkins-bot:

[research/recommendation-api@master] Avoid duplicate slash in URLs for cxserver API

https://gerrit.wikimedia.org/r/1061662

Change #1061950 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: use cxserver api without trailing slash

https://gerrit.wikimedia.org/r/1061950

Change #1061950 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: use cxserver api without trailing slash

https://gerrit.wikimedia.org/r/1061950

@santhosh, thank you for pushing the fix to remove a trailing slash from the cxserver API. We have deployed the new image in staging, and both the section and translation recommendation endpoints are up and running:

# section recommendation endpoint
kevinbazira@deploy1003:~$ time curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/v1/translation/sections?source=en&target=es&seed=Apple&search_algorithm=morelike&count=3"
[{"source_title":"List of apple cultivars","target_title":"Anexo:Cultivares de Manzano","source_sections":["Table of apples","Cider apples","Rootstock cultivars","See also","References","Further reading"],"target_sections":["A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z","Referencias"],"present":{"References":"Referencias"},"missing":{"See also":"Véase también","Further reading":"Bibliografía","Table of apples":"Table of apples","Cider apples":"Cider apples","Rootstock cultivars":"Rootstock cultivars"}},{"source_title":"Malus","target_title":"Malus","source_sections":["Description","Subdivisions and species","Cultivation","Toxicity","Uses","Gallery","References","External links"],"target_sections":["Características","Taxonomía","Referencias","Bibliografía","Enlaces externos"],"present":{"References":"Referencias","External links":"Enlaces externos"},"missing":{"Description":"Descripción","Cultivation":"Cultivo","Uses":"Usos","Gallery":"Galería","Subdivisions and species":"Subdivisions and species","Toxicity":"Toxicity"}},{"source_title":"Pear","target_title":"Pyrus","source_sections":["Etymology","Description","History","Major species","Cultivation","Production","Storage","Uses","Nutrition","Cultural references","Gallery","See also","References","Further reading","External links"],"target_sections":["Etimología","Historia","Descripción","Distribución y hábitat","Taxonomía","Selección de especies","Usos","Referencias","BIbliografía","Enlaces externos"],"present":{"Etymology":"Etimología","Description":"Descripción","History":"Historia","Uses":"Usos","References":"Referencias","External links":"Enlaces externos"},"missing":{"Cultivation":"Cultivo","Production":"Producción","Nutrition":"Nutrición","Cultural references":"En la cultura popular","Gallery":"Galería","See also":"Véase también","Further reading":"Bibliografía","Major species":"Major species","Storage":"Storage"}}]
real	0m1.818s
user	0m0.009s
sys	0m0.003s

# translation recommendation endpoint
kevinbazira@deploy1003:~$ time curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[{"title":"Flamenco (apple)","pageviews":0,"wikidata_id":"Q19597233","rank":7.0,"langlinks_count":1},{"title":"Applecrab","pageviews":0,"wikidata_id":"Q19595924","rank":8.0,"langlinks_count":0},{"title":"Malus × zumi","pageviews":0,"wikidata_id":"Q5990804","rank":21.0,"langlinks_count":3}]
real	0m0.550s
user	0m0.007s
sys	0m0.007s

Thanks @kevinbazira. I also tested, LGTM.

From API consumer side, CX is ready to use this based on https://gerrit.wikimedia.org/r/1059549

It is ready to deploy to production.

Change #1062348 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: prod config for modernized rec-api

https://gerrit.wikimedia.org/r/1062348

Change #1062348 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: prod config for modernized rec-api

https://gerrit.wikimedia.org/r/1062348

@santhosh, thank you for the confirmation. @klausman, as shown in P67284, the rec-api has been deployed in prod. Previously, the internal endpoint used https://recommendation-api-ng.discovery.wmnet:31443/api/spec and now it works with https://recommendation-api-ng.discovery.wmnet:31443/docs. Following your example in T347263#9392641, I expected the external endpoint to work with
https://api.wikimedia.org/service/lw/recommendation/v1/docs but it returns {"detail":"Not Found"}. If the internal endpoint is working fine shouldn't the external endpoint work as well? What could I be missing?

This is likely caused by a missing / in the API GW config. I will prepare a patch in a moment. For illustration you can try curl https://api.wikimedia.org/service/lw/recommendation/v1docs, which will give you the right html body (but the page won't work since other requests fail, specifically openapi.json).

Change #1062368 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] api-gw/liftwing: add missing tailing `/` to path trim

https://gerrit.wikimedia.org/r/1062368

One thing to note is that with the current scheme (plus the slash-fixing patch above), we would have:

Naturally, this isn't very pretty with the double v1. The API GW config (https://gerrit.wikimedia.org/g/operations/deployment-charts/+/refs/changes/68/1062368/2/helmfile.d/services/api-gateway/values.yaml#247) is flexible, but I don't want to make it overly complex. What are people's thoughts on how this external API should look and how we should rewrite it? The prefix /service/lw/recommendation is something we definitely want to keep. So we migth change the config to allow for this:

@klausman, +1 on using https://api.wikimedia.org/service/lw/recommendation/ as the API GW endpoint. Considering the rec-api code takes care of versioning internally, we don't need to repeat it in the API GW config.

Currently https://api.wikimedia.org/service/lw/recommendation/v1api/v1/translation?source=en&target=fr&count=12&seed=Apple works.
An ideal api URL should be

https://api.wikimedia.org/service/lw/recommendation/v1/translation?source=en&target=fr&count=12&seed=Apple or
https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=12&seed=Apple

This is causing a production issue right now as CX is not able to show translation suggestions https://en.wikipedia.org/wiki/Special:ContentTranslation#suggestions as configured api is throwing 404

Change #1062368 merged by jenkins-bot:

[operations/deployment-charts@master] api-gw/liftwing: fix prefix/trim for rec-api isvc

https://gerrit.wikimedia.org/r/1062368

With change 1062368 merged and pushed, this should now work correctly.

@santhosh, apologies for the inconvenience. The modernized recommendation-api is now live in LiftWing production. It can be accessed through:
1.External endpoint:

$ curl "https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[{"title":"Flamenco (apple)","pageviews":0,"wikidata_id":"Q19597233","rank":7.0,"langlinks_count":1},{"title":"Applecrab","pageviews":0,"wikidata_id":"Q19595924","rank":9.0,"langlinks_count":0},{"title":"Plum pox","pageviews":0,"wikidata_id":"Q1788571","rank":19.0,"langlinks_count":5}]

2.Internal endpoint:

$ curl "https://recommendation-api-ng.discovery.wmnet:31443/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[{"title":"Flamenco (apple)","pageviews":0,"wikidata_id":"Q19597233","rank":7.0,"langlinks_count":1},{"title":"Applecrab","pageviews":0,"wikidata_id":"Q19595924","rank":8.0,"langlinks_count":0},{"title":"Malus × zumi","pageviews":0,"wikidata_id":"Q5990804","rank":22.0,"langlinks_count":3}]

3.Documentation:

• WMF API Gateway documentation
• Wikitech LiftWing documentation

Please let us know in case there are any edge cases we may have missed. :)

Change #1062685 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/extensions/ContentTranslation@master] Use the updated recommendation API from liftwing

https://gerrit.wikimedia.org/r/1062685

Change #1062685 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Use the updated recommendation API from liftwing

https://gerrit.wikimedia.org/r/1062685

Change #1062696 had a related patch set uploaded (by KartikMistry; author: Santhosh):

[mediawiki/extensions/ContentTranslation@wmf/1.43.0-wmf.18] Use the updated recommendation API from liftwing

https://gerrit.wikimedia.org/r/1062696

Change #1062697 had a related patch set uploaded (by KartikMistry; author: Santhosh):

[mediawiki/extensions/ContentTranslation@wmf/1.43.0-wmf.17] Use the updated recommendation API from liftwing

https://gerrit.wikimedia.org/r/1062697

Change #1062697 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.43.0-wmf.17] Use the updated recommendation API from liftwing

https://gerrit.wikimedia.org/r/1062697

Change #1062696 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.43.0-wmf.18] Use the updated recommendation API from liftwing

https://gerrit.wikimedia.org/r/1062696

Mentioned in SAL (#wikimedia-operations) [2024-08-14T13:16:32Z] <kartik@deploy1003> Started scap sync-world: Backport for [[gerrit:1062697|Use the updated recommendation API from liftwing (T371465)]]

Mentioned in SAL (#wikimedia-operations) [2024-08-14T13:18:58Z] <kartik@deploy1003> kartik: Backport for [[gerrit:1062697|Use the updated recommendation API from liftwing (T371465)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-08-14T13:25:09Z] <kartik@deploy1003> Finished scap sync-world: Backport for [[gerrit:1062697|Use the updated recommendation API from liftwing (T371465)]] (duration: 08m 37s)

Mentioned in SAL (#wikimedia-operations) [2024-08-14T13:25:52Z] <kartik@deploy1003> Started scap sync-world: Backport for [[gerrit:1062696|Use the updated recommendation API from liftwing (T371465)]]

Mentioned in SAL (#wikimedia-operations) [2024-08-14T13:28:08Z] <kartik@deploy1003> kartik: Backport for [[gerrit:1062696|Use the updated recommendation API from liftwing (T371465)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-08-14T13:33:44Z] <kartik@deploy1003> Finished scap sync-world: Backport for [[gerrit:1062696|Use the updated recommendation API from liftwing (T371465)]] (duration: 07m 51s)

Change #1063360 had a related patch set uploaded (by Santhosh; author: Santhosh):

[research/recommendation-api@master] Fix the broken /docs url in production

https://gerrit.wikimedia.org/r/1063360

A minor issue we need to fix before closing the ticket is to fix broken documentation at https://api.wikimedia.org/service/lw/recommendation/docs

While attempting to fix this, I found that we can probably avoid the path trimming in values.html.

If API_PREFIX=/service/lw/recommendation/api/ env value is set, fastapi application will serve https://server:port/service/lw/recommendation/api/v1/translation?source..

That means, the liftwing URL part can be directly passed to recommendation api without trimming. Patch https://gerrit.wikimedia.org/r/1063360 implements this.

This could be a little bit tricky to deploy, so please be careful while deploying.

Change #1063360 merged by jenkins-bot:

[research/recommendation-api@master] Fix the broken /docs url in production

https://gerrit.wikimedia.org/r/1063360

Change #1063729 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: set API_PREFIX in rec-api

https://gerrit.wikimedia.org/r/1063729

Change #1063729 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: set API_PREFIX in rec-api

https://gerrit.wikimedia.org/r/1063729

@santhosh, the fix to the broken documentation has been deployed following your recommendation of setting the API_PREFIX:

• it runs fine in staging: https://phabricator.wikimedia.org/P67380
• but fails in prod: https://phabricator.wikimedia.org/P67381

Change #1064021 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/puppet@production] httpbb: add post deployment tests for the rec-api endpoint

https://gerrit.wikimedia.org/r/1064021

@klausman thank you for updating the trim prefix in the API GW config. @santhosh, the modernized recommendation-api is now up and running in LiftWing production:
1.Interactive documentation:

• https://api.wikimedia.org/service/lw/recommendation/api/docs

2.Grafana dashboard:

• https://grafana.wikimedia.org/goto/m57NC9CSR?orgId=1

3.External endpoint:

$ time curl "https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[{"title":"Flamenco (apple)","pageviews":0,"wikidata_id":"Q19597233","rank":7.0,"langlinks_count":1},{"title":"Applecrab","pageviews":0,"wikidata_id":"Q19595924","rank":9.0,"langlinks_count":0},{"title":"Malus × zumi","pageviews":0,"wikidata_id":"Q5990804","rank":19.0,"langlinks_count":3}]
real	0m2.838s
user	0m0.048s
sys	0m0.008s

4.Internal endpoint:

$ time curl "https://recommendation-api-ng.discovery.wmnet:31443/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[{"title":"Flamenco (apple)","pageviews":0,"wikidata_id":"Q19597233","rank":7.0,"langlinks_count":1},{"title":"Applecrab","pageviews":0,"wikidata_id":"Q19595924","rank":9.0,"langlinks_count":0},{"title":"Malus × zumi","pageviews":0,"wikidata_id":"Q5990804","rank":19.0,"langlinks_count":3}]
real	0m1.001s
user	0m0.013s
sys	0m0.000s

Change #1064021 merged by Klausman:

[operations/puppet@production] httpbb: add post deployment tests for the rec-api endpoint

https://gerrit.wikimedia.org/r/1064021