Page MenuHomePhabricator
Create Task
Maniphest T204950

Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
Open, LowPublic0 Estimated Story Points
Actions

Description

As part of the Wikistats 2 project we have developed the Edit Data Lake (see https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits). The Edit Data Lake is a denormalized data store that is the best dataset we have had to date to answer questions about content and contributors. At this time this data is only available for the WMF in the private hadoop cluster.
This is the parent task for all the work to make the Data Lake data available on our public cloud infrastructure for our community at large; the more accessible that data is, the more impact it can have.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT204950 Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
 DeclinedOttomataT204951 Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users
 ResolvedNoneT199674 eqiad | (3) Labs Data Lake hardware
Unknown Object (Task)
 Resolved• CmjohnsonT207192 rack/setup/install an-worker10[78-95].eqiad.wmnet
 ResolvedOttomataT207194 rack/setup/install cloudvirtan100[1-5].eqiad.wmnet
 ResolvedOttomataT203115 Create .deb package for Presto
 DeclinedNoneT207205 Set up 3 Ganeti VMs for datalake cloud analytics Hadoop cluster
 ResolvedOttomataT207321 Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster
 ResolvedJAllemandouT208614 Presto on Cloud Platform Design Document
ResolvedAndrewT211599 Set up 5 decidated VMs on the cloudvirtan hardware in the cloud-analytics project
 DeclinedAndrewT211634 Increase quota in cloud-analytics project for zookeeper nodes
 DeclinedOttomataT211640 Grafana, icinga, prometheus in cloud-analytics project
 DuplicateNoneT214043 Make edit data lake data available as a snapshot on dump hosts
 Resolved• CmjohnsonT225128 Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN.
 OpenNoneT215858 Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema
 ResolvedfnegriT280152 Mitigate breaking changes from the new Wiki Replicas architecture
 Resolved• BstormT260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture
 Resolved• BstormT260843 Set up roles for new wiki replicas layout
 ResolvedMarosteguiT265135 wikireplicas: Define MW sections per host
 ResolvedMarosteguiT267090 Productionize clouddb10[13-20]
 ResolvedMarosteguiT268312 Deploy labsdbuser and views to new clouddb hosts
 Resolved• BstormT269200 Create end-user accounts on the new clouddb hosts
 Resolved• BstormT269620 maintain-dbusers doesn't close connections right on harvest-replicas
 ResolvedMoritzMuehlenhoffT268725 Include mail on standard_packages.pp
 ResolvedMarosteguiT268742 Test upgrading sanitarium hosts to Buster + 10.4
 ResolvedMarosteguiT272008 Move wikireplicas under the new sanitarium hosts (db1154, db1155)
 Resolved• CmjohnsonT272125 Memory errors on clouddb1019
 ResolveddcaroT272127 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL **
 ResolvedMarosteguiT280492 Upgrade all sanitarium masters to 10.4 and Buster
 ResolvedRequestwiki_willyT281794 decommission db1082.eqiad.wmnet
 ResolvedRequest• CmjohnsonT281959 decommission db1074.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282079 decommission db1079.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282093 decommission db1087.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282096 decommission db1085.eqiad.wmnet
 DeclinedNoneT267376 Set up IP addresses for the new wiki replicas setup
 Resolved• KormatT268098 Make innodb_change_buffering configurable
 ResolvedArielGlennT261145 Enable access for wmcs-admins to run wmcs-prefixed cookbooks on cumin hosts
 Resolved• BstormT264254 Prepare Quarry for multiinstance wiki replicas
 Declined• BstormT267989 Do some checks of how many Quarry queries will break in a multiinstance environment
 Resolved• BstormT272723 Create a way to sample wikireplicas usage data
 Resolved• BstormT266266 Improve dry-run behavior of wmcs.wikireplicas.add_wiki cookbook
 Resolved• JhernandezT268498 Feedback from Quarry and PAWS users, and other wiki editors affected by the new architecture
 Resolved• razziT269211 Convert labsdb1012 from multi-source to multi-instance
 ResolvedMarosteguiT270473 Ensure InnoDB is compressed on the new clouddb hosts
 ResolvedMilimetricT274690 Update sqoop to work with multi-instance clouddb1021 mariadb host
 Resolved• BstormT281287 Now that labsdb1012 is clouddb1021, connect it to maintain-dbusers
 Resolved• BstormT269399 Set up a way to persist non-default number of wikireplicas connections across all instances
 Resolved• BstormT260511 Parametrize wmf-pt-kill so it can connect to different sockets
 Resolved• BstormT274044 Fix systemd and possibly logrotate around the wmf-pt-kill service for multi-instance wikireplicas
 Resolved• BstormT271476 Iron out issues in the proxy structure for multi-instance wikireplicas
 ResolvedAndrewT272553 Enable the creation of the svc.wikimedia.cloud wikireplicas addresses in Designate
 Resolved• BstormT272720 Allocate service IPs for new wikireplicas setup
 Resolved• ayounsiT273248 wikireplicas last-minute infra work to discuss / resolve
 Resolved• JhernandezT272523 Early testing of the new Wiki Replicas multi-instance architecture
 ResolvedMarosteguiT273593 Clean up heartbeat table on clouddb hosts
 ResolvedRagesossT278983 New Replica returns different data than old Replica for query using `change_tag` table
 Resolved• BstormT276284 Establish a working setup for PAWS with multi-instance wikireplicas
 Resolved• BstormT278252 Make alias for tools.db.svc.wikimedia.cloud
 Resolved• BstormT281732 Check into the configuration, cause and usefulness of memory alerts for multiinstance replicas
 OpenNoneT267992 Provide mechanism to detect name clashed media between Commons and a Local project, without needing to join tables across wiki-db's
 OpenNoneT268240 Provide a mechanism for detecting duplicate files in commons and a local wiki
 OpenNoneT268242 Provide a mechanism for detecting duplicate files in enwiki and another wikipedia
 OpenNoneT268244 Provide a mechanism for accessing the names of image files on Commons when querying another wiki
 OpenNoneT258511 Data Lake incremental Data Updates
OpenNoneT231938 Get "edits hourly" on a daily basis
 ResolvedMilimetricT258532 [SPIKE] Prototype of incremental updates for mediawiki history for simplewiki , including reverts using apache hudi
DeclinedNoneT262205 Need for new event-type - `user_create` and `user_rename`
 ResolvedJAllemandouT262256 Test hudi and Iceberg as an incremental update system using 2 mediawiki-history snapshots
 DeclinedNoneT262260 Make hudi work with Hive
 ResolvedJAllemandouT262261 Check whether mediawiki production event data is equivalent to mediawiki-history data over a month
ResolvedMilimetricT215001 Revisions missing from mediawiki_revision_create
 ResolvedNoneT280538 Capture rev_is_revert event data in a stream different than mediawiki.revision-create
 DeclinedMilimetricT263055 Add log entry details to page and user events in EventBus
 OpenNoneT240387 MW REST API Historical Data Endpoint Needs
 ResolvedMilimetricT241184 Design Document that proposes an alternative architecture for historic data endpoints
 DeclinedNoneT241185 Flink Spike
OpenOttomataT370354 [SPIKE] Experiment with approaches for a incremental updates of MediaWiki data in the Data Lake
 OpenNoZeroDayT373144 [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally

Event Timeline

• Nuria created this task.Sep 20 2018, 1:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2018, 1:44 PM
• Nuria renamed this task from Edit Data Lake available in labs: Mediawiki history snapshots available in SQL data store to Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to clod (labs) users.Sep 20 2018, 1:45 PM
• Nuria added a subtask: T204537: Document on wikitech presto choice for data lake .
• Nuria removed a subtask: T204537: Document on wikitech presto choice for data lake .
• Nuria updated the task description. (Show Details)Sep 20 2018, 1:53 PM
• Nuria updated the task description. (Show Details)
Milimetric renamed this task from Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to clod (labs) users to Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users.Sep 24 2018, 3:49 PM
Milimetric triaged this task as Medium priority.
Milimetric removed a project: Analytics-Kanban.
Milimetric moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
• Tbayer subscribed.Sep 27 2018, 1:09 AM
Milimetric raised the priority of this task from Medium to High.Oct 18 2018, 5:36 PM
Milimetric added a project: Analytics-Kanban.
Milimetric set the point value for this task to 0.
Milimetric removed a project: Analytics.Oct 19 2018, 4:15 PM
Jdforrester-WMF subscribed.Nov 7 2018, 5:48 PM
nshahquinn-wmf subscribed.Feb 22 2019, 1:39 PM
Milimetric moved this task from Parent Tasks to Paused on the Analytics-Kanban board.Mar 4 2019, 4:58 PM
Milimetric added a project: Cloud-Services.Mar 4 2019, 5:08 PM
Milimetric merged a task: T169572: Provide mediawiki history data to Cloud Services users.
Milimetric added subscribers: MusikAnimal, zhuyifei1999.
Ottomata added a subtask: T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..Jun 27 2019, 5:26 PM
• Nuria edited projects, added Analytics; removed Analytics-Kanban.Jul 22 2019, 8:49 PM
elukey closed subtask T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. as Resolved.Sep 27 2019, 2:17 PM
• Nuria edited projects, added Analytics-Kanban; removed Analytics.Nov 19 2019, 12:25 AM
• Nuria moved this task from Paused to Parent Tasks on the Analytics-Kanban board.
Milimetric edited projects, added Analytics; removed Analytics-Kanban.Jan 2 2020, 6:21 PM
Milimetric subscribed.

De-prioritizing until cloud infrastructure can support monitoring similar to what we can do in production.

Milimetric moved this task from Smart Tools for Better Data to Deprioritized on the Analytics board.Jan 2 2020, 6:21 PM
Milimetric lowered the priority of this task from High to Low.Aug 31 2020, 5:01 PM
Milimetric moved this task from Deprioritized to Smart Tools for Better Data on the Analytics board.

This needs further consideration.

Milimetric closed subtask T204951: Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users as Declined.Aug 31 2020, 5:03 PM
nshahquinn-wmf removed a subtask: T204953: Data quality issues in the mediawiki_history data stream.Oct 30 2020, 10:32 AM
• nskaggs subscribed.Nov 18 2020, 8:24 PM
• Jhernandez subscribed.Nov 19 2020, 11:51 AM
taavi edited projects, added Data-Services; removed Cloud-Services.Oct 9 2021, 5:17 PM
bd808 moved this task from Backlog to Datasets on the Data-Services board.Apr 12 2022, 11:24 PM
ttaylor subscribed.Dec 1 2022, 10:25 PM
Aklapper added a project: Data-Engineering-Icebox.Feb 10 2023, 5:44 PM
TBurmeister subscribed.Oct 31 2023, 5:15 PM
taavi moved this task from Datasets to Backlog on the Data-Services board.Oct 22 2024, 5:21 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptOct 22 2024, 5:21 PM
taavi subscribed.Oct 22 2024, 5:22 PM

Sorry to poke an many years old ticket.. but what still needs to happen here? All of the subtasks have been resolved already.

Ottomata subscribed.Oct 22 2024, 5:34 PM

@taavi many tickets were declined for complexity reasons, but we have new ways of potentially doing this. It needs to be prioritized though, so if you have desires/needs, please escalate them through https://www.mediawiki.org/wiki/Data_Platform_Engineering/Intake_Process

taavi closed this task as Declined.Oct 24 2024, 2:45 PM

That seems unnecessarily complicated when I just want to get old stale tasks off of the Data-Services board. I'll just close this instead, if someone is interested in getting this through the process they're free to re-open this.

Ottomata reopened this task as Open.Oct 24 2024, 3:26 PM
Ottomata removed projects: cloud-services-team, Data-Services, Analytics.

I'd like to keep this open, but I will remove the Data-Services tag. It is something we would really like to do.

Ottomata added a subtask: T215858: Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema.Tue, Nov 12, 3:25 PM
Ottomata added a subtask: T258511: Data Lake incremental Data Updates .
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits