Page MenuHomePhabricator

mobrovac (Marko Obrovac)
Disabled

Projects

User Details

User Since
Dec 16 2014, 7:40 PM (519 w, 3 d)
Roles
Disabled
IRC Nick
mobrovac
LDAP User
Mobrovac
MediaWiki User
Mobrovac [ Global Accounts ]

Recent Activity

Dec 16 2019

mobrovac lowered the priority of T240518: Some jobs are not being processed / are processed slowly from Unbreak Now! to High.

Lowering the priority as we are out of the woods for the time being. Jobs are being processed now, but it will take a while to catch up with current events (at least 24h form now).

Dec 16 2019, 8:10 PM · Sustainability (Incident Followup), SDC-Statements (Machine-vision-depicts), MachineVision, Product-Infrastructure-Team-Backlog-Deprecated, Structured-Data-Backlog, MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Platform Team Workboards (Clinic Duty Team), WMF-JobQueue, SRE

Dec 13 2019

mobrovac placed T137350: Create general guidelines & processes to ensure thorough fault testing of services up for grabs.
Dec 13 2019, 12:12 PM · Sustainability (Incident Followup), serviceops, Platform Engineering
mobrovac placed T106374: DOI redirect handler for iopscience is very slow/ potentially causes citoid to behave oddly. up for grabs.
Dec 13 2019, 12:11 PM · Platform Team Legacy (Watching / External), Services (watching), Citoid
mobrovac placed T105780: Create a doc explaining the SLA between services and the monitoring tool up for grabs.
Dec 13 2019, 12:11 PM · Observability-Alerting, Platform Team Legacy (Later), Services (later), Documentation, SRE, Service-Architecture
mobrovac placed T215220: Zlib spec.yaml example produces a bad request response up for grabs.
Dec 13 2019, 12:11 PM · Math, User-Physikerwelt, Mathoid
mobrovac placed T122825: Service Ownership and Maintenance up for grabs.
Dec 13 2019, 12:11 PM · Platform Engineering, TechCom, User-mobrovac, SRE
mobrovac placed T235437: RESTBase/RESTRouter/service-runner rate limiting plans up for grabs.
Dec 13 2019, 12:11 PM · Platform Engineering (Icebox), service-runner, User-mobrovac, Services (doing), Core Platform Team Initiatives (RESTBase Split (CDP2)), serviceops, Kubernetes, Service-deployment-requests, SRE
mobrovac placed T239616: scap3 rollback does not depool servers before rolling back up for grabs.
Dec 13 2019, 12:10 PM · Platform Engineering (Icebox), Scap

Dec 10 2019

mobrovac moved T218395: Create RestrictionStore service from External Code Review Needed to External Code Review In Progress on the Platform Team Workboards (Clinic Duty Team) board.
Dec 10 2019, 12:47 PM · Patch-For-Review, MW-1.37-notes (1.37.0-wmf.23; 2021-09-13), Platform Team Workboards (MW Expedition), MW-1.35-notes (1.35.0-wmf.18; 2020-02-04), Core Platform Team Initiatives (Decoupling (CDP2))
mobrovac moved T230800: Improve Rdbms/LoadBalance and its LoadMonitor logic from External Code Review Needed to External Code Review In Progress on the Platform Team Workboards (Clinic Duty Team) board.

FYI, I already +1'ed the patch, but @jcrespo raised some questions on the PS that I think should be addressed before merging.

Dec 10 2019, 12:41 PM · MW-1.35-notes (1.35.0-wmf.23; 2020-03-10), Platform Team Workboards (Clinic Duty Team), Patch-For-Review, Performance-Team (Radar), MediaWiki-libs-Rdbms

Dec 9 2019

mobrovac added a project to T238917: Empty reply for pages with Special characters in french REST API: Platform Team Workboards (Clinic Duty Team).
Dec 9 2019, 11:25 AM · RESTBase-API, Platform Engineering (Icebox)
mobrovac added a project to T239713: Citoid is logging all request / response headers as separate fields: Platform Team Workboards (Clinic Duty Team).
Dec 9 2019, 10:44 AM · Observability-Logging, observability, Wikimedia-Logstash, Platform Engineering (Icebox), serviceops, SRE, Citoid
mobrovac added a project to T239090: Restbase logging indexing conflict on 'res' and 'body' logging fields: Platform Team Workboards (Clinic Duty Team).
Dec 9 2019, 10:43 AM · SRE Observability, observability, Platform Team Workboards (Clinic Duty Team), Wikimedia-Logstash, RESTBase
mobrovac removed a member for WMF-JobQueue: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a watcher for WMF-JobQueue: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a watcher for Trending-Service: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a member for Trending-Service: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a member for Texvcjs: mobrovac.
Dec 9 2019, 10:03 AM
mobrovac removed a member for TechCom-RFC: mobrovac.
Dec 9 2019, 10:02 AM
mobrovac removed a member for TechCom: mobrovac.
Dec 9 2019, 10:02 AM
mobrovac removed a watcher for TechCom: mobrovac.
Dec 9 2019, 10:02 AM
mobrovac removed a member for Services: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a watcher for Services: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a member for service-template-node: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a watcher for service-template-node: mobrovac.
Dec 9 2019, 10:01 AM
mobrovac removed a member for service-runner: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a watcher for service-runner: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a member for Service-deployment-requests: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a watcher for Service-deployment-requests: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a member for Service-Architecture: mobrovac.
Dec 9 2019, 10:00 AM
mobrovac removed a watcher for Service-Architecture: mobrovac.
Dec 9 2019, 9:59 AM
mobrovac removed a member for SCB: mobrovac.
Dec 9 2019, 9:59 AM
mobrovac removed a watcher for SCB: mobrovac.
Dec 9 2019, 9:59 AM
mobrovac removed a member for Scap: mobrovac.
Dec 9 2019, 9:58 AM
mobrovac archived RESTBase-Usecase.
Dec 9 2019, 9:58 AM
mobrovac removed a member for RESTBase-Usecase: mobrovac.
Dec 9 2019, 9:57 AM
mobrovac removed a watcher for RESTBase-Usecase: mobrovac.
Dec 9 2019, 9:57 AM
mobrovac removed a member for RESTBase-release-1.0: mobrovac.
Dec 9 2019, 9:55 AM
mobrovac archived RESTBase-release-1.0.
Dec 9 2019, 9:55 AM
mobrovac removed a watcher for RESTBase-release-1.0: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a member for RESTBase-Cassandra: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a watcher for RESTBase-Cassandra: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a member for RESTBase-architecture: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a watcher for RESTBase-architecture: mobrovac.
Dec 9 2019, 9:54 AM
mobrovac removed a watcher for RESTBase-API: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a member for RESTBase-API: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a member for RESTBase: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a watcher for RESTBase: mobrovac.
Dec 9 2019, 9:53 AM
mobrovac removed a member for Push-Notification-Service: mobrovac.
Dec 9 2019, 9:52 AM
mobrovac removed a member for MediaWiki-Vagrant: mobrovac.
Dec 9 2019, 9:52 AM
mobrovac removed a member for MediaWiki-Containers: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a watcher for MediaWiki-Containers: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a member for Mathoid: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a watcher for Mathoid: mobrovac.
Dec 9 2019, 9:51 AM
mobrovac removed a member for HyperSwitch: mobrovac.
Dec 9 2019, 9:50 AM
mobrovac removed a watcher for HyperSwitch: mobrovac.
Dec 9 2019, 9:50 AM
mobrovac removed a member for Prod-Kubernetes: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a watcher for Prod-Kubernetes: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a member for Event-Platform: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a watcher for Event-Platform: mobrovac.
Dec 9 2019, 9:49 AM
mobrovac removed a member for Dumps-Rewrite: mobrovac.
Dec 9 2019, 9:48 AM
mobrovac removed a watcher for Dumps-Rewrite: mobrovac.
Dec 9 2019, 9:48 AM
mobrovac removed a member for Dependency-Tracking: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a watcher for Dependency-Tracking: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a member for Citoid: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a watcher for Citoid: mobrovac.
Dec 9 2019, 9:46 AM
mobrovac removed a member for ChangeProp: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a watcher for ChangeProp: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a member for Cassandra: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a watcher for Cassandra: mobrovac.
Dec 9 2019, 9:45 AM
mobrovac removed a member for Release Pipeline: mobrovac.
Dec 9 2019, 9:44 AM

Dec 6 2019

mobrovac assigned T239877: After deploy of 1.35.0-wmf.8 to group1, surge of "Cannot access the database: Unknown error" to Anomie.
Dec 6 2019, 11:57 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Platform Team Workboards (Clinic Duty Team), DBA, Wikimedia-production-error
mobrovac added a project to T239877: After deploy of 1.35.0-wmf.8 to group1, surge of "Cannot access the database: Unknown error": Platform Team Workboards (Clinic Duty Team).
Dec 6 2019, 11:56 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Platform Team Workboards (Clinic Duty Team), DBA, Wikimedia-production-error

Dec 5 2019

mobrovac closed T239466: Possible to circumvent title-blacklist (CVE-2019-19709) as Resolved.
Dec 5 2019, 1:27 PM · Security, Platform Team Workboards (Clinic Duty Team), MediaWiki-Action-API, TitleBlacklist
mobrovac moved T236624: Unable to restore page: IncompleteRevisionException for some revisions from Waiting for Review to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Dec 5 2019, 1:26 PM · MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), Platform Team Workboards (Clinic Duty Team), Technical-Debt, Multi-Content-Revisions, Wikimedia-production-error, MediaWiki-Core-Revision-backend
mobrovac moved T238921: MCR: Include all slots in XML dumps per default from Waiting for Review to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.

@daniel Gerrit 552564 which is awaiting review depends on a WIP patch, so moving back to Ready.

Dec 5 2019, 1:24 PM · MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), MW-1.35-release, Core Platform Team Initiatives (MCR Schema Migration), Platform Team Workboards (Clinic Duty Team), Dumps-Generation
mobrovac moved T239428: API edit on page with non-resolvable redirect and redirect=1 creates page with invalid title from Waiting for Review to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Dec 5 2019, 1:18 PM · MW-1.33-notes, MW-1.31-release-notes, MW-1.32-notes, MW-1.34-notes, MW-1.35-notes (1.35.0-wmf.5; 2019-11-05), Platform Team Workboards (Clinic Duty Team), MediaWiki-Action-API
mobrovac added a comment to T206283: Failed deferred updates should be queued as jobs if possible (Deadlock from LinksUpdate in WikiPage::updateCategoryCounts).

@Krinkle @aaron Gerrit 522234 has been merged. As for Gerrit 497537, two things are missing:
(i) Amir had a concern on the PS that went unaddressed;

OK. I've signed the Gerrit change back over to @aaron.

Dec 5 2019, 1:04 PM · MW-1.35-notes (1.35.0-wmf.24; 2020-03-17), Platform Team Workboards (Clinic Duty Team), MediaWiki-Page-derived-data, Performance-Team, Wikimedia-production-error

Dec 4 2019

mobrovac moved T232866: VirtualRESTServiceClient returns status code 0 on Docker from External Code Review Needed to External Code Review Completed on the Platform Team Workboards (Clinic Duty Team) board.
Dec 4 2019, 8:14 PM · MW-1.34-notes, MW-1.33-notes, MW-1.32-notes, MW-1.35-notes (1.35.0-wmf.10; 2019-12-10), MW-1.31-release-notes, Patch-For-Review, MediaWiki-libs-HTTP, Platform Team Workboards (Clinic Duty Team)
mobrovac closed T239120: Services downstream from RESTBase should not separately alert due to RESTBase production monitoring timeouts as Declined.

RESTBase (and other services) are alerting because we want to be sure that we catch problems visible by external users. While it does sometimes introduce false positives (the inverse is true too - RB alerts if back-end services alert), it is better to have more alerts than none due to convoluted logic or fixtures.

Dec 4 2019, 7:53 PM · Product-Infrastructure-Team-Backlog-Deprecated, Platform Engineering, Recommendation-API, Page Content Service, Wikifeeds
mobrovac added a comment to T239806: Parsoid/PHP errors.

I should note that most of the errors appearing there originate from ChangePropagation, so they seem like genuine ones (given that Parsoid/JS don't spit out an error for 99% of these).

Dec 4 2019, 4:37 PM · User-brennen, Parsoid-PHP
mobrovac closed T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page as Resolved.
Dec 4 2019, 3:31 PM · Platform Team Workboards (Clinic Duty Team), Parsoid-PHP, RESTBase
mobrovac moved T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page from Inbox to Waiting for deployment on the Platform Team Workboards (Clinic Duty Team) board.
Dec 4 2019, 1:17 PM · Platform Team Workboards (Clinic Duty Team), Parsoid-PHP, RESTBase
mobrovac claimed T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page.

PR #1226 adds the title component.

Dec 4 2019, 1:07 PM · Platform Team Workboards (Clinic Duty Team), Parsoid-PHP, RESTBase
mobrovac added a subtask for T229015: Tracking: Direct live production traffic at Parsoid/PHP: T239806: Parsoid/PHP errors.
Dec 4 2019, 12:37 PM · User-notice-archive, Platform Engineering, User-WDoran, Parsoid-PHP
mobrovac added a parent task for T239806: Parsoid/PHP errors: T229015: Tracking: Direct live production traffic at Parsoid/PHP.
Dec 4 2019, 12:37 PM · User-brennen, Parsoid-PHP
mobrovac triaged T239806: Parsoid/PHP errors as High priority.
Dec 4 2019, 12:37 PM · User-brennen, Parsoid-PHP

Dec 3 2019

mobrovac updated subscribers of T239643: Bugs in PHP port of LanguageConverter.

Mirroring had to be disabled due to too many errors being emitted by Parsoid/PHP. Once the fixes for it have been deployed, RESTBase can de re-set up to mirror the traffic. I have prepared everything for it, all that needs to be done is to deploy restbase with scap deploy -f from deploy1001.

Dec 3 2019, 1:23 PM · MediaWiki-Language-converter, Chinese-Sites, Parsoid-PHP
mobrovac added a comment to T239643: Bugs in PHP port of LanguageConverter.

RESTBase is now mirroring all html2html requests to Parsoid/PHP.

Dec 3 2019, 12:46 PM · MediaWiki-Language-converter, Chinese-Sites, Parsoid-PHP

Dec 2 2019

mobrovac moved T239607: MediaWiki::restInPeace: transaction round 'MediaWiki\Linter\RecordLintJob::run' still running from Inbox to Done on the Platform Team Workboards (Clinic Duty Team) board.
Dec 2 2019, 3:36 PM · Platform Team Workboards (Clinic Duty Team), serviceops
mobrovac closed T239607: MediaWiki::restInPeace: transaction round 'MediaWiki\Linter\RecordLintJob::run' still running as Resolved.
Dec 2 2019, 3:36 PM · Platform Team Workboards (Clinic Duty Team), serviceops
mobrovac claimed T239616: scap3 rollback does not depool servers before rolling back.
Dec 2 2019, 3:17 PM · Platform Engineering (Icebox), Scap
mobrovac triaged T239394: Can't rename the translatable page mw:Global_templates/Draft_spec as Medium priority.

Yes, the time-out is there on purpose. All jobs share it. We cannot increase it on a per-job basis, though, only for job runners as a whole. Why does a page rename in this instance take so long in the first place? I think this is the angle that should be looked at.

Dec 2 2019, 12:43 PM · User-DannyS712, Platform Team Workboards (Clinic Duty Team), WMF-JobQueue, MediaWiki-extensions-Translate

Nov 26 2019

mobrovac closed T234266: Cannot read property 'stored' of undefined as Resolved.

As of the RESTBase deploy, these errors do not appear any more. Resolving.

Nov 26 2019, 9:08 AM · Platform Team Workboards (Clinic Duty Team), RESTBase, Parsoid
mobrovac updated the task description for T229015: Tracking: Direct live production traffic at Parsoid/PHP.
Nov 26 2019, 7:57 AM · User-notice-archive, Platform Engineering, User-WDoran, Parsoid-PHP
mobrovac closed T237143: Log warning: Duplicate get(): "officewiki:echo:seen:message:time:{n}" fetched 2 times as Resolved.
Nov 26 2019, 7:36 AM · Notifications, MediaWiki-libs-BagOStuff, Growth-Team, Platform Team Workboards (Clinic Duty Team)
mobrovac triaged T239190: WikiFeeds in Labs not running as High priority.
Nov 26 2019, 7:12 AM · Product-Infrastructure-Team-Backlog-Deprecated (Kanban), Wikifeeds

Nov 25 2019

mobrovac moved T234266: Cannot read property 'stored' of undefined from Doing(WIP:5) to Waiting for Review on the Platform Team Workboards (Clinic Duty Team) board.

Ok found the bug. The problem was that the filter routine that executed the actual conversion was loaded twice. PR #1219 fixes this.

Nov 25 2019, 5:45 PM · Platform Team Workboards (Clinic Duty Team), RESTBase, Parsoid
mobrovac added a comment to T234266: Cannot read property 'stored' of undefined.

I've established that it's definitely RESTBase's fault: for some reason, when the revision is provided, it asks for the conversion twice, but the second time it provides the content returned to it by Parsoid for the first conversion request. I'm still investigating why and how that happens. To be continued.

Nov 25 2019, 12:34 PM · Platform Team Workboards (Clinic Duty Team), RESTBase, Parsoid
mobrovac added a comment to T238954: html2wt: Missing implementation of 'ParsoidFetchTemplateData' to fetch templatedata.

@ssastry note that if you let it ride the train, that means setting switching back for two weeks (this week + next week for the train). Ideally, you may want to put the patch up for SWAT as soon as it gets merged into master (either Tuesday or next Monday).

Nov 25 2019, 8:12 AM · MW-1.35-notes (1.35.0-wmf.8; 2019-11-26), Parsoid-PHP