OK.

I've added this to Tech News #36 which will go out on September 2 (no issue on Monday, all the usual writers are travelling and no one else picked up the torch). I've also posted on Project Chat on Wikidata as that is fairly soon.

In general, I recommend:

• Posting on the Village Pump, to give a heads up, at least a couple of weeks prior to the read-only period. It would be good if this could be done for all wikis.
• Mentioning it in Tech News. (This is being done.)
  • We can follow up in Tech News later, especially for September 10, 24 and 26. The wikis on September 24 are a) few and b) not Commons and Wikidata that are used by other wikis. If we want to remind the wikis on September 17, that's probably best done individually, outside of Tech News.
• Set up banner maybe 30–60 minutes before the read-only period. This can be done by asking the communities or by a CentralNotice banner (ping @Trizek-WMF).

This is a routine by now, and works well, so the above should be enough.

I'll be OoO for two and half weeks starting tomorrow, so someone else will have to take this up from here on.

@Marostegui Something that would be helpful would be if we could have a simplified technical explanation (aimed at fairly non-technical audience) explaining why we need the read-only periods, so that we could explain to the communities what's happening and why they can't edit. Would you have the time to do that?

(If you feel uncomfortable writing for a non-technical audience: we can help with editing, once we're back.)

Sure @Johan, let me know if this is good enough or you need more or less detail.

There is an on-site maintenance at our primary datacenter, where the primary database masters are located. The maintenance will be specifically on those racks, and it involves plugging and unplugging servers from one power source to another, whilst all our servers have redundant power supplies and they will most likely not lose power at anytime, accidents can occur and if that happens, the affected server can go off.
Our primary database masters are the ones that get the edits, and replicate them to other hosts (replicas) where reads happen.

If any of our primary database masters lose power, we don't only lose the ability to edit until it is restored (it can take a few minutes for a server to boot up) but we could run into data corruptions (due to that abrupt crash).
To avoid running into any of those unexpected scenarios, we prefer to switch our master to a host located in a rack that won't be affected by this maintenance.
The reason the read-only time is required, is because we have to change the configuration to make mediawiki point to the new host, and that needs to be done while there are not edits happening to avoid the possibility of a split brain (an edit happening at the exact moment the master switch change is being propagated to all our mediawiki servers).

@Marostegui Thanks! The one additional detail some people keep asking about where it'd be great if I could give them a good answer would be what in our setup creates this problem, when they can't remember it happening for other websites.

That is a broader discussion.
Essentially, the architecture we have at the moment (both, systems and MW related) is thought to have only one active master at the time. So only one host receives writes at the time. Having more than one host allowing this, is something that we are discussing but requires lots of changes on both, our system's architecture and on MW code itself.

There is a huge number of organizations out there with a similar architecture, and the key is to do this switchover as fast as possible so your users don't get impacted that much. We have done tremendous improvements to be able to do this under 2 minutes (and we the latest changes we are aiming to do it under 1 minute now).

There is one thing to keep in mind here, and it is the fact that we do announce our read-only times, even though they are very short, some other organizations don't do so, and they let their users experiment an error, which is usually fixed the next time the write is attempted if the process is fast. So the fact that sometimes the feeling is that other websites doesn't suffer from this problem doesn't mean it is not there, it is handled in a different way.
In some organizations showing errors during a minute (or less) is fine and it is assumed as part of regular maintenance. We, however, prefer to announce our read-only times to make sure users and bots are aware so we don't create unexpected inconveniences.

Hope this helps!

@Marostegui is this task only to coordinate our team support?

In T230788#5451661, @Elitre wrote:

@Marostegui is this task only to coordinate our team support?

Yes :-)
The technical bits are on different tickets

Thanks for confirming! In this case, it'd be helpful to follow the structure suggested at https://office.wikimedia.org/wiki/Community_Relations#Public_requests_(standard) - we more or less know what it is that you may need from us, but more details for everyone else could also help - and thanks for the email headsup BTW!

In T230788#5451733, @Elitre wrote:

Thanks for confirming! In this case, it'd be helpful to follow the structure suggested at https://office.wikimedia.org/wiki/Community_Relations#Public_requests_(standard) - we more or less know what it is that you may need from us, but more details for everyone else could also help - and thanks for the email headsup BTW!

I have edited the task with that template.

Here is the Great Banners Matrix.
Each cell is a separate banner. Since we have on banner template and I don't want to multiply it, I will setup each banner once at the time.

Wikis that can't be covered:

• arbcom
• boardgovcom
• board
• beta
• chairwiki
• chapcomwiki
• checkuserwiki
• collabwiki
• donate
• electcomwiki
• exec
• noboard_chapters,
• nostalgia
• foundation
• fixcopyright
• grants
• id_internal
• incubator
• internal
• legalteam
• login
• map_bms
• movementroles
• nds_nl
• nyc
• pa_us
• office
• otrs
• ombudsmen
• quality
• searchcom
• strategy
• stewards
• spcom
• ten
• techconduct
• iegcom
• projectcom
• all test wikis
• transitionteam
• usability
• vote
• wg_en

We can skip them, since they don't seem to have a lot of trafic, or being used by people who know how to use them. So the locked database message would be enough, with no prior warning.

Wikimania wikis are read-only, except wikimania.wikimedia.org. But since Wikimania is over, we can expect to only rely on the locked database message.

@Znotch190711 why do you consider this as UBN?

Banner set for Sept 10. Will be displayed from 04:30 to 05:30 UTC.

s8 (wikidata) has been done today:
read-only start: Tue Sep 10 05:00:47 UTC 2019
read-only stop: Tue Sep 10 05:02:14 UTC 2019

Total read-only time: 1 minute 27 seconds.

Thank you for the update here @Marostegui , much appreciated!

3 banners set for September 17.

I'm warming up for the 24th! :)

Just for the record, I realised that the banner for today on itwiki is wrong:

A breve verrà svolta della manutenzione tecnica. 17 settembre - 05:00 AM UTC - 05:30 AM UTC
Durante tale intervallo potresti non riuscire a salvare alcuna modifica. (dalle 15:00 alle 15:15 UTC del 19 marzo 2019)

So the first part is correct, 17th Sept from 05:00AM-05:30AM UTC, but the second part looks wrong (19 march from 15:00 to 15:15 UTC).
Just saying it here in case you get some messages from people about it as it is a past date :-)

s2 switchover is done
read-only start: 05:00:44
read-only stop: 05:01:34

Total read-only time: 50 seconds.

In T230788#5497842, @Marostegui wrote:

Just for the record, I realised that the banner for today on itwiki is wrong:

A breve verrà svolta della manutenzione tecnica. 17 settembre - 05:00 AM UTC - 05:30 AM UTC
Durante tale intervallo potresti non riuscire a salvare alcuna modifica. (dalle 15:00 alle 15:15 UTC del 19 marzo 2019)

So the first part is correct, 17th Sept from 05:00AM-05:30AM UTC, but the second part looks wrong (19 march from 15:00 to 15:15 UTC).
Just saying it here in case you get some messages from people about it as it is a past date :-)

Thank you for noticing it! The original text does not have that "19 march from 15:00 to 15:15 UTC" part. It is an addition that passed the translation validation without being noticed. I will fix it to avoid that propagation for the next banners.

Banners set for tomorrow.

Concerning chapters and usergroups sites (*.wikimedia.org), the system for banners is automatically filtering languages. I don't know how much of those wikis will get the banner. However, most users on those wikis are experienced wikimedians and they will not be surprised to have the "database locked" maintenance message.

Surprisingly, Outreach is not eligible for a banner. But it is included if you target all wikis. (This system is not optimal at all.) I've warned the wiki.

s3 was done successfully.

read only start: 05:10:14 UTC AM
read only stop: 05:13:08 UTC AM

total read only time: 2 minutes 54 seconds.

We had a slightly longer read only time compared to the all the previous ones due to some issues with the way we set read-only, those will be followed up at T233679

Thanks for your support!

My pleasure!

s4 was done:

Read-only start: 05:00:51
Read-only stop: 05:01:42
Total read-only time: 51 seconds

In T230788#5428358, @Johan wrote:

@Marostegui Something that would be helpful would be if we could have a simplified technical explanation (aimed at fairly non-technical audience) explaining why we need the read-only periods, so that we could explain to the communities what's happening and why they can't edit. Would you have the time to do that?

Hi @Johan and @Marostegui,

I’m reading this a bit late but your explanations have clarified things for me and removed my doubts that these read-only periods and WMF’s datacenters are managed seriously (those doubts I expressed on the translators mailing list at the end of July, with a quite rude tone, sorry for that). So if I understand well, the architecture model which is used is so that:

• each wiki always has a unique primary database master (active master) which is the only server to receive write operations (which can be more than 100 in one minute on en.wp if counting only the changes on pages) and (probably, but it might be a setting) does not deal with any read operations;
• this primary database master transfers the changes to replica hosts which deal with read operations (which are certainly much more numerous than the write operations);
• it’s not difficult to change the active master (so I guess it’s a replica which ceases to communicate with readers and instead communicates with writers), but this can lead to losing some edits (so maybe this part could be fixed specifically);
• thus, a maintenance operation on the active master will be likely to imply a read-only period.

And the MediaWiki software does not allow to have several active masters at a time (so that one could save the other).

I’m pleased to see that my work as a translator can lead me to improve my skills in computing too. Thank you!