labsdb1005 (mysql) maintenance for reimage
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Feb 6 2017, 5:29 PM

Description

Scheduled for 15 February.

Announcement:

Hello!

Tools DB and Labs Postgres DB will be undergoing maintenance on 15 Feb
2017 for about 6 hours starting at 1700 UTC and will be unreachable
for some of the duration. Most users shouldn't experience issues if
their code reconnects properly when the server stops accepting
connections (we'll failover to slaves when doing maintenance). Some
tables will not be available for a short period of time, but the tool
owners of those tables have already been notified (see
https://phabricator.wikimedia.org/T127164 ). We'll try to minimize
downtime as much as possible.

We will be upgrading the operating system from Ubuntu Precise to
Debian Jessie in preparation for EOL of Ubuntu Precise (in April
2017). We'll also take this opportunity to upgrade Tools DB to Mariadb

All data should be preserved in this migration. Follow
https://phabricator.wikimedia.org/T123731 for more information!

Thanks!

Planning:

Switchover labsdb1005 to labsdb1004
Backup labsdb1005 somewhere
Reimage labsdb1005
Recover data
Switchover labsdb1004 back to labsdb1005

Details

Subject	Repo	Branch	Lines +/-
toolsdb: Increase innodb log file size to 500M (1 GB total)	operations/puppet	production	+1 -1
linux-host-entries: Remove precise from labsdb1005	operations/puppet	production	+0 -2
Upgrade toolsdb master to mariadb10	operations/puppet	production	+2 -5

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dzahn	T123525 reduce amount of remaining Ubuntu 12.04 (precise) systems in production
Resolved	jcrespo	T123731 Migrate labsdb1005/1006/1007 to jessie
Resolved	Marostegui	T157358 labsdb1005 (mysql) maintenance for reimage

Event Timeline

jcrespo created this task.Feb 6 2017, 5:29 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 6 2017, 5:29 PM

jcrespo updated the task description. (Show Details)Feb 6 2017, 5:31 PM

jcrespo updated the task description. (Show Details)

Adding user notice. In theory, no end users should be affected, but if some tools have not been properly programmed to reconnect, they could have issues during and after upgrade.

Johan moved this task from To Triage to Announce in next Tech/News on the User-notice board.Feb 8 2017, 12:52 PM

Johan moved this task from Announce in next Tech/News to In current Tech/News draft on the User-notice board.Feb 8 2017, 1:02 PM

Liuxinyu970226 subscribed.Feb 10 2017, 1:01 AM

jcrespo moved this task from Triage to Pending comment on the DBA board.Feb 14 2017, 3:04 PM

jcrespo moved this task from Pending comment to In progress on the DBA board.

jcrespo mentioned this in T123731: Migrate labsdb1005/1006/1007 to jessie.Feb 14 2017, 8:05 PM

RandomDSdevel awarded a token.Feb 15 2017, 1:32 AM

04:23 < yuvipanda> marostegui: jynus I can verify that I can access labsdb1004 from tools, so no need to massage VLANs or firewalls                                                                                                                                
04:24 < yuvipanda> I do find that it has less databases than 1005 tho. not sure if that's expected                                                                                                                                                      
04:46 < yuvipanda> jynus: marostegui https://gerrit.wikimedia.org/r/#/c/337775/ will switch the aliases we ask people to use to labsdb1004 from 1005                                                                                                                                                                                                                                                                                              
07:15 < marostegui> yuvipanda: https://phabricator.wikimedia.org/P4935                                                                                                                                                                                             
07:16 < marostegui> I guess it is not too worrying

After a chat with Jaime we have moved those old databases in labsdb1005 to: labsdb1005:/srv/tmp/old_dbs .
They didn't have any data, and only one of them contained some tables (empty, only the frm).
So now labsdb1004 and labsdb1005 have the same amount of databases.

For the backup data:

es1017 looks like a good candidate:

marostegui@es1017:~$ df -hT /srv
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs    11T  5.1T  5.9T  47% /srv

labsdb1005 totalsize for /srv is less than 950G:

root@labsdb1005:~# df -hT /srv/labsdb/
Filesystem                        Type  Size  Used Avail Use% Mounted on
/dev/mapper/labsdb1005--vg-labsdb ext4  8.0T  943G  7.1T  12% /srv/labsdb
root@labsdb1005:~# df -hT /srv/postgres/
Filesystem                          Type  Size  Used Avail Use% Mounted on
/dev/mapper/labsdb1005--vg-postgres ext4  4.0T   67M  4.0T   1% /srv/postgres

Change 337881 had a related patch set uploaded (by Jcrespo):
Upgrade toolsdb master to mariadb10

https://gerrit.wikimedia.org/r/337881

gerritbot added a project: Patch-For-Review.Feb 15 2017, 5:39 PM

Change 337881 merged by Jcrespo:
Upgrade toolsdb master to mariadb10

https://gerrit.wikimedia.org/r/337881

Change 337911 had a related patch set uploaded (by Marostegui):
linux-host-entries: Remove precise from labsdb1005

https://gerrit.wikimedia.org/r/337911

Change 337911 merged by Jcrespo:
linux-host-entries: Remove precise from labsdb1005

https://gerrit.wikimedia.org/r/337911

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['labsdb1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702152016_jynus_16501.log.

Completed auto-reimage of hosts:

['labsdb1005.eqiad.wmnet']

and were ALL successful.

Change 337990 had a related patch set uploaded (by Jcrespo):
toolsdb: Increase innodb log file size to 500M (1 GB total)

https://gerrit.wikimedia.org/r/337990

Dzahn awarded a token.Feb 15 2017, 10:06 PM

Change 337990 abandoned by Jcrespo:
toolsdb: Increase innodb log file size to 500M (1 GB total)

https://gerrit.wikimedia.org/r/337990

This is mostly done, no major incidents- servers where only in read-only for a few seconds before and after the maintenance, for switchover to the slave.

We will monitor in case someone has an issue.

Johan moved this task from In current Tech/News draft to Recently announced in Tech/News on the User-notice board.Feb 16 2017, 11:22 AM

jcrespo mentioned this in T138517: mysqldump is timing out preventing all tables from being included in the dump.Feb 20 2017, 2:23 PM

Marostegui awarded a token.Feb 20 2017, 3:21 PM

Johan moved this task from Recently announced in Tech/News to Already announced/Archive on the User-notice board.Feb 22 2017, 5:42 PM

Lokal_Profil mentioned this in T158911: Docker setup is broken because of InnoDB and FULLTEXT.Feb 24 2017, 9:10 AM