[00:10:29] !log Wikimedia SAL messages now available on Mastodon! https://fosstodon.org/@wikimedia_sal [00:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:33:33] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [00:35:21] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [01:39:11] PROBLEM - Maps tiles generation on icinga1001 is CRITICAL: CRITICAL: 90.42% of data under the critical threshold [5.0] https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1 [02:28:48] (03PS1) 10Jeroen De Dauw: Add our wiki blog to EN planet [puppet] - 10https://gerrit.wikimedia.org/r/560154 [04:03:31] (03PS2) 10Andrew Bogott: toolforge: Move package 'fish' from shell_environ to exec_environ [puppet] - 10https://gerrit.wikimedia.org/r/560008 (https://phabricator.wikimedia.org/T241290) (owner: 10Zhuyifei1999) [04:11:08] (03CR) 10Andrew Bogott: [C: 03+2] toolforge: Move package 'fish' from shell_environ to exec_environ [puppet] - 10https://gerrit.wikimedia.org/r/560008 (https://phabricator.wikimedia.org/T241290) (owner: 10Zhuyifei1999) [04:26:10] 10Operations, 10Traffic: Add more detailed instructions to the "sec-advice" page - https://phabricator.wikimedia.org/T241309 (10DannyS712) [04:32:46] (03PS2) 10Andrew Bogott: labs: send integration alerts to the team [puppet] - 10https://gerrit.wikimedia.org/r/559858 (owner: 10Hashar) [04:34:47] (03CR) 10Andrew Bogott: [C: 03+2] "I hope everyone enjoys the dings!" [puppet] - 10https://gerrit.wikimedia.org/r/559858 (owner: 10Hashar) [05:58:07] !log volker-e@deploy1001 Started deploy [design/style-guide@9aa0b3d]: Deploy design/style-guide: [05:58:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:58:14] !log volker-e@deploy1001 Finished deploy [design/style-guide@9aa0b3d]: Deploy design/style-guide: (duration: 00m 07s) [05:58:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:15] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [08:09:03] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [10:03:15] 10Operations, 10ops-eqiad, 10cloud-services-team (Kanban): cloudvirt1013 apparent power loss - https://phabricator.wikimedia.org/T241315 (10Andrew) @Cmjohnson or or @Jclark-ctr, I don't want to prompt another reboot but could you give the power cables on cloudvirt1013 an extra push and see if you notice any... [10:21:16] 10Operations, 10Traffic: cp3051 crashed - https://phabricator.wikimedia.org/T241306 (10ema) Thanks @volans for taking care of this. >>! In T241306#5759025, @Volans wrote: > Nothing in `racadm`, checked both `getsel` and `lclog view`. Nothing in syslog & co. Just like all other crashes tracked in T238305 :-/ N... [12:40:40] (03PS1) 10Andrew Bogott: nova firstboot: add a few setup steps to firstboot.sh [puppet] - 10https://gerrit.wikimedia.org/r/560206 (https://phabricator.wikimedia.org/T239347) [12:50:35] (03PS2) 10Andrew Bogott: nova firstboot: add a few setup steps to firstboot.sh [puppet] - 10https://gerrit.wikimedia.org/r/560206 (https://phabricator.wikimedia.org/T181375) [15:02:47] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [15:06:21] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [16:04:05] (03PS1) 10Reedy: Enable WebAuthn extension on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560227 [16:34:01] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [16:35:49] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [16:44:23] !log Trying another Mastodon account. Hopefully running at bot at @wikimedia_sal@mastodon.social is acceptable use for that server. [16:44:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:45:34] (03PS1) 10Subscriptshoe9: T150618,Upload HD Logo for fawikivoyage, jawikiquote&cywikiquote,Google Code-in 2019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560314 [16:45:38] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560314 (owner: 10Subscriptshoe9) [16:48:43] !log Mastodon feed moved to https://mastodon.social/@wikimedia_sal (T52109) [16:48:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:48:59] T52109: Add Mastodon support to stashbot - https://phabricator.wikimedia.org/T52109 [17:00:05] cool! [17:11:39] (03Abandoned) 10Subscriptshoe9: T150618,Upload HD Logo for fawikivoyage, jawikiquote&cywikiquote,Google Code-in 2019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560314 (owner: 10Subscriptshoe9) [17:22:49] (03PS1) 10Subscriptshoe9: T150618,Upload HD Logo for fawikivoyage, jawikiquote&cywikiquote,Google Code-in 2019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560318 [17:38:35] * Reedy wonders why they're mentioning GCI in hte commit summary [17:39:21] idk :) [17:42:51] Something tells me they haven't done the beginner task for making a change in gerrit [17:43:17] No content, and WIP... [17:43:21] * Reedy waits before posting https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines [17:43:32] (03Abandoned) 10Subscriptshoe9: T150618,Upload HD Logo for fawikivoyage, jawikiquote&cywikiquote,Google Code-in 2019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560318 (owner: 10Subscriptshoe9) [17:43:35] Reedy: that happens when you click "create commit" via Gerrit's site [17:43:43] I know [17:43:51] But the commit summary doesn't [17:43:56] And they've not marked it ready [17:44:04] They've already already abandond one commit for whatever reason [17:45:06] I'll leave a note via the GCI site [18:01:54] (03PS1) 10BryanDavis: Toolforge: pass X-Forwared-* headers from front proxy to apps [puppet] - 10https://gerrit.wikimedia.org/r/560323 (https://phabricator.wikimedia.org/T241310) [18:21:56] (03PS2) 10BryanDavis: Toolforge: pass X-Forwared-* headers from front proxy to apps [puppet] - 10https://gerrit.wikimedia.org/r/560323 (https://phabricator.wikimedia.org/T241310) [18:33:15] (03CR) 10BryanDavis: Toolforge: pass X-Forwared-* headers from front proxy to apps (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/560323 (https://phabricator.wikimedia.org/T241310) (owner: 10BryanDavis) [18:59:29] (03CR) 10BryanDavis: [C: 03+1] "I have applied the k8s config change already with a manual edit of the cluster state. I have also tested the urlproxy change on tools-prox" [puppet] - 10https://gerrit.wikimedia.org/r/560323 (https://phabricator.wikimedia.org/T241310) (owner: 10BryanDavis) [21:01:49] PROBLEM - CirrusSearch eqiad 95th percentile latency on graphite1004 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [1000.0] https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelId=19&fullscreen&orgId=1&var-cluster=eqiad&var-smoothing=1 [21:04:34] (03PS1) 10MarcoAurelio: Enable subpages for the main namespaces on ge.wikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560339 (https://phabricator.wikimedia.org/T241329) [21:09:58] 10Operations, 10ops-codfw: codfw: rack/setup/install puppetmaster2003.codfw.wmnet - https://phabricator.wikimedia.org/T239732 (10Papaul) [21:17:45] (03PS2) 10MarcoAurelio: Enable subpages for the main namespace on ge.wikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560339 (https://phabricator.wikimedia.org/T241329) [21:20:23] 10Operations, 10ops-codfw, 10Wikimedia-Logstash: rack/setup/install logstash202[6-9].codfw.wmnet - https://phabricator.wikimedia.org/T240882 (10Papaul) [21:30:25] (03PS1) 10Ladsgroup: Revert "Add a bit for forcing LC caching backend in cli mode" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560341 [22:31:52] (03CR) 10Ladsgroup: mediawiki: Use mediawiki::errorpage instead of a php7-fatal-error.php.erb (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/539203 (https://phabricator.wikimedia.org/T113114) (owner: 10Ladsgroup) [22:37:03] (03PS6) 10Ladsgroup: mediawiki: Use mediawiki::errorpage instead of a php7-fatal-error.php.erb [puppet] - 10https://gerrit.wikimedia.org/r/539203 (https://phabricator.wikimedia.org/T113114) [23:14:01] 10Operations, 10ops-codfw, 10DBA: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) [23:16:37] 10Operations, 10ops-codfw, 10DBA: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) @Marostegui in the racking proposal, you reaqusted that es2020 be racked in A2 and es2022 in C2. A2 and C2 are 10G racks so we need to move those 2 servers into a 1G rac... [23:17:14] 10Operations, 10ops-codfw, 10DBA: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) p:05Triage→03Normal [23:32:31] 10Operations, 10ops-codfw: codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org - https://phabricator.wikimedia.org/T241337 (10Papaul) [23:33:35] 10Operations, 10ops-codfw: codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org - https://phabricator.wikimedia.org/T241337 (10Papaul) p:05Triage→03Normal [23:37:39] PROBLEM - Check for VMs leaked by the nova-fullstack test on cloudcontrol1003 is CRITICAL: 10 instances in the admin-monitoring project https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting%23Nova-fullstack [23:44:54] PROBLEM - Check for VMs leaked by the nova-fullstack test on cloudcontrol1003 is CRITICAL: 10 instances in the admin-monitoring project https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting%23Nova-fullstack [23:55:37] RECOVERY - Check for VMs leaked by the nova-fullstack test on cloudcontrol1003 is OK: 0 instances in the admin-monitoring project https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting%23Nova-fullstack