Page MenuHomePhabricator
Create Task
Maniphest T319434

service implementation tracking: arclamp1001.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

The following steps are needed to replace webperf1004 with the new arclamp1001 hardware:

  • add rsync config (rsync::quickdatacopy) to allow syncing data from webperf1004 to arclamp1001
  • add xenon user/group: groupadd --system xenon --gid 1001 && useradd --system --gid xenon --uid 498 xenon && install -d -o xenon -g xenon /srv/xenon
  • do the initial rsync and validate filesystem permissions

Coordianate a one hour window with Performance team for the actual migration:

  • disable Puppet on webperf1004
  • mask the excimer-k8s-log.service, excimer-k8s-wall-log.service, excimer-log.service, excimer-wall-log.service services on webperf1004 (that's why we need to disable puppet, otherwise it would unmask them)
  • we also need to mask some timers used for the sync: arclamp_generate_metrics.timer, arclamp_compress_logs.timer, arclamp_generate_svgs.timer
  • rsync again to catch up with the latest changes
  • apply the webperf::profiling_tools role to arclamp1001
  • Change arclamp_host in Hiera to point to arclamp1001
  • Extend scap/dsh config to also deploy to arclamp1001
  • puppet needs to remain disabled on webperf1004 before it's decommed, so best to set downtime for a week

Details

SubjectRepoBranchLines +/-
Decom webperf[12]004operations/puppetproduction+0 -14
webperf: update rsync source/dest with the new realityoperations/puppetproduction+2 -2
Move arclamp to arclamp1001operations/puppetproduction+2 -3
site: apply webperf::profiling_tools to arclamp1001operations/puppetproduction+8 -8
arclamp: move to EnvironmentFile for generate/compress jobsoperations/puppetproduction+22 -8
webperf: use rsync::quickdatacopy for arclamp dataoperations/puppetproduction+9 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Oct 5 2022, 3:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2022, 3:35 PM
jijiki moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.Oct 10 2022, 8:55 AM
Dzahn changed the task status from Open to Stalled.Oct 12 2022, 7:51 PM
RobH unsubscribed.Oct 13 2022, 3:18 PM
Krinkle renamed this task from service implementation tracking: webperf1005.eqiad.wmnet to service implementation tracking: arclamp1001.eqiad.wmnet.Oct 15 2022, 10:21 PM
Krinkle updated the task description. (Show Details)
LSobanski removed Dzahn as the assignee of this task.Oct 18 2022, 3:48 PM
LSobanski subscribed.

Doesn't look like collab, unassigning from Daniel.

Dzahn added a comment.Nov 9 2022, 5:56 PM

per T316223#8381863 serviceops-core is taking this over

Dzahn changed the task status from Stalled to Open.Nov 9 2022, 5:57 PM
Dzahn mentioned this in T316223: Expand RAM on arclamp hosts and move them to baremetal.Nov 9 2022, 6:02 PM
aaron subscribed.Dec 19 2022, 8:05 PM
In T319434#8383918, @Dzahn wrote:

per T316223#8381863 serviceops-core is taking this over

Let us know if there is anything you need from the perf team.

Dzahn added a subscriber: akosiaris.Dec 19 2022, 11:14 PM

@aaron CCing @akosiaris Depending how you want to look at it this is either a subtask and unblocks or a duplicate of T316223. Also see T316223#8383941, T316223#8185277. Cheers

akosiaris edited projects, added observability; removed serviceops.Dec 20 2022, 9:18 AM
akosiaris added a subscriber: lmata.

Actually, per 28f86674054b7 observability has taken over arclamp from serviceops. It aligns more closely with their area of expertise and focus than serviceops and we are thankful. I forgot to update T316223 to account for this, doing so now. Also adding @lmata as the best person now to take this over.

MoritzMuehlenhoff added a comment.Dec 20 2022, 9:24 AM
In T319434#8480708, @akosiaris wrote:

Actually, per 28f86674054b7 observability has taken over arclamp from serviceops. It aligns more closely with their area of expertise and focus than serviceops and we are thankful. I forgot to update T316223 to account for this, doing so now. Also adding @lmata as the best person now to take this over.

I had synched up with Leo on this: Since I migrated webperf* to Bullseye, I will pair up with someone from o11y to move these to their new baremetal home.

lmata added a subscriber: fgiunchedi.Dec 20 2022, 3:51 PM
In T319434#8480722, @MoritzMuehlenhoff wrote:
In T319434#8480708, @akosiaris wrote:

Actually, per 28f86674054b7 observability has taken over arclamp from serviceops. It aligns more closely with their area of expertise and focus than serviceops and we are thankful. I forgot to update T316223 to account for this, doing so now. Also adding @lmata as the best person now to take this over.

I had synched up with Leo on this: Since I migrated webperf* to Bullseye, I will pair up with someone from o11y to move these to their new baremetal home.

indeed, we have a q3 placeholder for this. /cc @fgiunchedi

Dzahn unsubscribed.Dec 20 2022, 7:02 PM
MoritzMuehlenhoff triaged this task as Medium priority.Jan 12 2023, 9:31 AM
MoritzMuehlenhoff updated the task description. (Show Details)
fgiunchedi added a project: User-fgiunchedi.Jan 13 2023, 9:31 AM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 13 2023, 11:21 AM
gerritbot added a comment.Jan 13 2023, 2:38 PM

Change 879813 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] webperf: use rsync::quickdatacopy for arclamp data

https://gerrit.wikimedia.org/r/879813

gerritbot added a project: Patch-For-Review.Jan 13 2023, 2:38 PM

Change 879814 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] arclamp: move to EnvironmentFile for generate/compress jobs

https://gerrit.wikimedia.org/r/879814

gerritbot added a comment.Jan 16 2023, 1:02 PM

Change 879813 merged by Filippo Giunchedi:

[operations/puppet@production] webperf: use rsync::quickdatacopy for arclamp data

https://gerrit.wikimedia.org/r/879813

gerritbot added a comment.Jan 16 2023, 1:05 PM

Change 879814 merged by Filippo Giunchedi:

[operations/puppet@production] arclamp: move to EnvironmentFile for generate/compress jobs

https://gerrit.wikimedia.org/r/879814

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2023, 1:30 PM
gerritbot added a comment.Jan 18 2023, 9:05 AM

Change 881352 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: apply webperf::profiling_tools to arclamp1001

https://gerrit.wikimedia.org/r/881352

gerritbot added a project: Patch-For-Review.Jan 18 2023, 9:05 AM

Change 881353 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Move arclamp to arclamp1001

https://gerrit.wikimedia.org/r/881353

Stashbot added a comment.Jan 18 2023, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-18T09:49:37Z] <godog> start migration from webperf1004 to arclamp1001 - T319434

gerritbot added a comment.Jan 18 2023, 9:55 AM

Change 881352 merged by Filippo Giunchedi:

[operations/puppet@production] site: apply webperf::profiling_tools to arclamp1001

https://gerrit.wikimedia.org/r/881352

gerritbot added a comment.Jan 18 2023, 10:12 AM

Change 881353 merged by Filippo Giunchedi:

[operations/puppet@production] Move arclamp to arclamp1001

https://gerrit.wikimedia.org/r/881353

fgiunchedi updated the task description. (Show Details)Jan 18 2023, 1:13 PM
gerritbot added a comment.Jan 18 2023, 4:23 PM

Change 881449 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] webperf: update rsync source/dest with the new reality

https://gerrit.wikimedia.org/r/881449

gerritbot added a comment.Jan 18 2023, 4:57 PM

Change 881449 merged by Filippo Giunchedi:

[operations/puppet@production] webperf: update rsync source/dest with the new reality

https://gerrit.wikimedia.org/r/881449

gerritbot added a comment.Jan 19 2023, 10:39 AM

Change 881569 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Decom webperf[12]004

https://gerrit.wikimedia.org/r/881569

gerritbot added a comment.Jan 19 2023, 10:58 AM

Change 881569 merged by Filippo Giunchedi:

[operations/puppet@production] Decom webperf[12]004

https://gerrit.wikimedia.org/r/881569

ops-monitoring-bot added a comment.Jan 19 2023, 11:08 AM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: webperf1004.eqiad.wmnet

  • webperf1004.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
fgiunchedi closed this task as Resolved.Jan 19 2023, 11:11 AM
fgiunchedi claimed this task.

This is completed -- arclamp is hosted on arclamp1001 and webperf1004 has been decom'd

fgiunchedi updated the task description. (Show Details)Jan 19 2023, 11:26 AM
lmata awarded a token.Jan 20 2023, 2:13 PM
lmata moved this task from Backlog to Radar on the SRE board.
lmata added a project: SRE Observability (FY2022/2023-Q3).
lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q3) board.Jan 20 2023, 2:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits