Page MenuHomePhabricator

[Commons Impact Metrics] Create a new AQS service with all the endpoints
Closed, ResolvedPublic5 Estimated Story Points

Description

This task requires that the API design is done T358679, and that the test data is in the corresponding AQS test environment T358715.
We should create a new AQS service with the endpoints defined in the API design.
Before deploying, we need T358707 to be finished!

Tasks:

  • Create a new repo for the service.
  • Copy paste all code from another service.
  • Modify all code to implement the new service.
  • entities
  • data layer
  • logic layer
  • handler layer
  • main
  • unit tests
  • integration tests
  • ...?
  • Make sure all tests pass
  • QA (should this be another task?)
  • Code review and deploy
  • Runbook
  • SLO

Definition of done:

  • The new service is queryable from AQS.
  • It serves Commons Impact Metrics data.

Event Timeline

@mforns please add a subtask for new service deployment and tag Data-Platform-SRE so they can prioritize the the work.

Gehel triaged this task as High priority.Mar 20 2024, 9:02 AM
mforns set the point value for this task to 34.Mar 21 2024, 2:15 PM
Gehel subscribed.

SRE specific work is tracked on T360531, removing SRE from this task.

Milimetric changed the point value for this task from 34 to 21.Apr 4 2024, 11:11 AM

I've broken this down into subtasks but I'm keeping it as something between an epic and an actual task. It's coordinating and has all the acceptance criteria, it was just too big. So I'll leave the other two subtasks on the boards while I'm on vacation and put this in paused. This can be resumed whenever you'd like to continue work on coordinating and deployment.

Milimetric changed the point value for this task from 21 to 5.Apr 15 2024, 4:16 PM

Test status: QA PASS

Pageviews per category monthly : PASS

code snippet:

import requests

from fake_useragent import UserAgent

ua = UserAgent()

url_full = 'http://localhost:8096/metrics/commons-analytics/pageviews-per-category-monthly/Abu_Bakr_medallion_in_Hagia_Sophia/shallow/all-wikis/20230501/20231201'

header = {"accept": "application/json",
          "user-agent": ua.chrome}

response = requests.get(url_full, headers=header)

print(response.status_code)

print(response.json())

print(response.headers)
  • Tested with valid endpoint. Status code: 200
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category request yield proper validation message
{
   "detail":"The date(s) you used are valid, but we either do not have data for those date(s), or the category you asked for is not loaded yet. Please check documentation for more information",
   "method":"get",
   "status":404,
   "title":"Not Found",
   "type":"about:blank",
   "uri":"/metrics/commons-analytics/pageviews-per-category-monthly/Abu_Bakr_medallion_in_Hagia_Sophi/shallow/all-wikis/20230501/20231201"
}
  • Invalid category scope yields proper validation message. Status code: 400
{
   "detail":"category scope should be equal to one of the allowed values: [shallow,deep]",
   "method":"get",
   "status":400,
   "title":"Bad Request",
   "type":"about:blank",
   "uri":"/metrics/commons-analytics/pageviews-per-category-monthly/Abu_Bakr_medallion_in_Hagia_Sophia/shallo/all-wikis/20230501/20231201"
}
  • Invalid wiki request yield validations error. Status Code: 404
{
   "detail":"The date(s) you used are valid, but we either do not have data for those date(s), or the category you asked for is not loaded yet. Please check documentation for more information",
   "method":"get",
   "status":404,
   "title":"Not Found",
   "type":"about:blank",
   "uri":"/metrics/commons-analytics/pageviews-per-category-monthly/Abu_Bakr_medallion_in_Hagia_Sophia/shallow/all-wiki/20230501/20231201"
}
  • Invalid start time yields proper validation
{
   "detail":"start timestamp is invalid, must be a valid date in YYYYMMDD format",
   "method":"get",
   "status":400,
   "title":"Bad Request",
   "type":"about:blank",
   "uri":"/metrics/commons-analytics/pageviews-per-category-monthly/Abu_Bakr_medallion_in_Hagia_Sophia/shallow/all-wikis/!20230501/20231201"
}
  • Invalid end time yields proper validation
{
   "detail":"end timestamp is invalid, must be a valid date in YYYYMMDD format",
   "method":"get",
   "status":400,
   "title":"Bad Request",
   "type":"about:blank",
   "uri":"/metrics/commons-analytics/pageviews-per-category-monthly/Abu_Bakr_medallion_in_Hagia_Sophia/shallow/all-wikis/20230501/!20231201"
}

Pageviews per media file monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid media file request yield proper validation message
  • Invalid wiki request yield validations error. Status Code: 404
  • Invalid start time yields proper validation
  • Invalid end time yields proper validation

Edits per Category Monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category request yield proper validation message. Status code : 404
  • Invalid category scope request yield validations error. Status Code: 400
  • Invalid edit type request yield validations error. Status Code: 400
  • Special character yields the right status code code. Status code : 404
  • Invalid start time yields proper validation. Status Code: 400
  • Invalid end time yields proper validation. Status Code: 400
  • Endtime before start yields proper validation. Status Code: 400

Edits per User Monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid username request yield proper validation message. Status code : 404
  • Invalid edit type request yield validations error. Status Code: 400
  • Special character yields the right status code code. Status code : 404
  • Invalid start time yields proper validation. Status Code: 400
  • Invalid end time yields proper validation. Status Code: 400
  • Endtime before start yields proper validation. Status Code: 400

Top pages per category monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category request yield proper validation message. Status code : 404
  • Invalid category scope request yield validations error. Status Code: 400
  • Invalid wiki type request yield validations error. Status Code: 404
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top wikis per category monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category request yield proper validation message. Status code : 404
  • Invalid category scope request yield validations error. Status Code: 400
  • Invalid wiki type request yield validations error. Status Code: 404
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top viewed categories monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category scope request yield validations error. Status Code: 400
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top pages per media file monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid media file yield proper validation message. Status code : 404
  • Invalid wiki type request yield validations error. Status Code: 404
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top wikis per media file monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid media file yield proper validation message. Status code : 404
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top viewed media files monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category request yield proper validation message. Status code : 404
  • Invalid category scope request yield validations error. Status Code: 400
  • Invalid wiki type request yield validations error. Status Code: 404
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top wikis by media file monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid media file yield proper validation message. Status code : 404
  • Invalid wiki type request yield validations error. Status Code: 404
  • Special character yields the right status code code. Status code : 404
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top edited categories monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category scope request yield validations error. Status Code: 400
  • Invalid edit type request yield validations error. Status Code: 400
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

Top edited Monthly : PASS

  • Tested with valid endpoint. Status code: 200. Results yield the right key-value pair
  • Content type in valid response is correct : 'Content-Type': 'application/json; charset=utf-8'
  • invalid route. Status code : 404, right content-type ( 'Content-Type': 'application/problem+json' )
  • Invalid category request yield validations error. Status Code: 404
  • Invalid category scope request yield validations error. Status Code: 400
  • Invalid edit type request yield validations error. Status Code: 400
  • Invalid Year yields proper validation. Status Code: 400
  • Invalid Month yields proper validation. Status Code: 400

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/87

Add repos/generated-data-platform/aqs/commons-impact-analytics to projects.json