[00:04:05] (03CR) 10Huji: [C: 04-1] "For now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/626851 (https://phabricator.wikimedia.org/T262689) (owner: 10Majavah) [00:04:16] PROBLEM - Check systemd state on netflow3001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:04:22] PROBLEM - Check systemd state on netflow5001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:05:24] PROBLEM - Check systemd state on netflow4001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:05:28] PROBLEM - Check systemd state on netflow2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:05:46] PROBLEM - Check systemd state on netflow1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:06:18] (03CR) 10Huji: [C: 04-1] "Further explanation: it is much easier for election admins (including T&S members) to set up the election in English. Only when the electi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/626851 (https://phabricator.wikimedia.org/T262689) (owner: 10Majavah) [01:09:24] PROBLEM - mediawiki-installation DSH group on parse2020 is CRITICAL: Host parse2020 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [01:14:10] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:34:52] PROBLEM - mediawiki-installation DSH group on parse2016 is CRITICAL: Host parse2016 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:35:20] PROBLEM - mediawiki-installation DSH group on parse2002 is CRITICAL: Host parse2002 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:35:20] PROBLEM - mediawiki-installation DSH group on parse2006 is CRITICAL: Host parse2006 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:38:14] PROBLEM - mediawiki-installation DSH group on parse2007 is CRITICAL: Host parse2007 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:43:02] PROBLEM - mediawiki-installation DSH group on parse2017 is CRITICAL: Host parse2017 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:43:20] PROBLEM - mediawiki-installation DSH group on parse2003 is CRITICAL: Host parse2003 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:46:04] PROBLEM - mediawiki-installation DSH group on parse2008 is CRITICAL: Host parse2008 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:49:50] PROBLEM - mediawiki-installation DSH group on parse2004 is CRITICAL: Host parse2004 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:52:30] PROBLEM - mediawiki-installation DSH group on parse2009 is CRITICAL: Host parse2009 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [02:58:48] PROBLEM - mediawiki-installation DSH group on parse2005 is CRITICAL: Host parse2005 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:11:20] PROBLEM - mediawiki-installation DSH group on parse2012 is CRITICAL: Host parse2012 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:15:44] PROBLEM - mediawiki-installation DSH group on parse2013 is CRITICAL: Host parse2013 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:16:22] PROBLEM - mediawiki-installation DSH group on parse2001 is CRITICAL: Host parse2001 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:18:14] PROBLEM - mediawiki-installation DSH group on parse2018 is CRITICAL: Host parse2018 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:20:32] PROBLEM - mediawiki-installation DSH group on parse2014 is CRITICAL: Host parse2014 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:22:36] PROBLEM - mediawiki-installation DSH group on parse2011 is CRITICAL: Host parse2011 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:22:36] PROBLEM - mediawiki-installation DSH group on parse2015 is CRITICAL: Host parse2015 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:25:28] PROBLEM - mediawiki-installation DSH group on parse2019 is CRITICAL: Host parse2019 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [03:29:32] PROBLEM - mediawiki-installation DSH group on parse2010 is CRITICAL: Host parse2010 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [04:13:17] 10Operations, 10Wikimedia-Mailing-lists: Figure out a way to sync old and new mailman - https://phabricator.wikimedia.org/T256539 (10Ladsgroup) >>! In T256539#6428881, @Legoktm wrote: > To be clear, we can import all of the legacy mailman 2 archives into hyperkitty, right? According to the documentation of mai... [04:23:29] 10Operations, 10Wikimedia-Mailing-lists: Upgrade to mailman 2.1.22+ - https://phabricator.wikimedia.org/T166944 (10Peachey88) 05Open→03Resolved We are now running 2.1.29 so it's been upgraded since the task was filled in 2017 mentioning 2.1.24, and is running the current debian stable. [04:23:32] 10Operations, 10Wikimedia-Mailing-lists, 10Upstream: Mailman: Consider hiding real list administrators email addresses - https://phabricator.wikimedia.org/T150164 (10Peachey88) [04:38:11] 10Operations, 10Wikimedia-Mailing-lists, 10Upstream: Mailman: Consider hiding real list administrators email addresses - https://phabricator.wikimedia.org/T150164 (10Legoktm) 05Open→03Resolved This seems to have been implemented now. I checked a few lists which say something similar to: > Wikimedia-l lis... [04:47:22] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/definition/{title} (retrieve en-wiktionary definitions for cat) is CRITICAL: Test retrieve en-wiktionary definitions for cat returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [04:49:22] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2001 is CRITICAL: Host parse2001 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2002 is CRITICAL: Host parse2002 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2003 is CRITICAL: Host parse2003 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2004 is CRITICAL: Host parse2004 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2005 is CRITICAL: Host parse2005 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2006 is CRITICAL: Host parse2006 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:41] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2007 is CRITICAL: Host parse2007 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:42] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2008 is CRITICAL: Host parse2008 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:43] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2009 is CRITICAL: Host parse2009 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:43] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2010 is CRITICAL: Host parse2010 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:44] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2011 is CRITICAL: Host parse2011 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:44] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2012 is CRITICAL: Host parse2012 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:45] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2013 is CRITICAL: Host parse2013 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:45] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2014 is CRITICAL: Host parse2014 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:45] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2015 is CRITICAL: Host parse2015 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:46] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2016 is CRITICAL: Host parse2016 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:46] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2017 is CRITICAL: Host parse2017 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:47] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2018 is CRITICAL: Host parse2018 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:48] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2019 is CRITICAL: Host parse2019 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:43:48] ACKNOWLEDGEMENT - mediawiki-installation DSH group on parse2020 is CRITICAL: Host parse2020 is not in mediawiki-installation dsh group Effie Mouzeli Servers are being put into service T247441 https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [05:44:25] 10Operations, 10serviceops, 10Patch-For-Review: move 20 new codfw parsoid servers (parse2*) into production - https://phabricator.wikimedia.org/T247441 (10jijiki) The DSH group alert fired on Saturday, I acked it:) [05:51:24] !log sudo -i depool mw2297 [05:51:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:30:36] ACKNOWLEDGEMENT - Apache HTTP on mw2297 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1310 bytes in 0.124 second response time Effie Mouzeli Host has been depooled https://wikitech.wikimedia.org/wiki/Application_servers [06:30:36] ACKNOWLEDGEMENT - PHP7 rendering on mw2297 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - header X-Powered-By: PHP/7. not found on http://en.wikipedia.org:80/wiki/Main_Page - 1310 bytes in 0.112 second response time Effie Mouzeli Host has been depooled https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:41:58] 10Operations, 10netops: logrotate failed on all netflow* hosts - https://phabricator.wikimedia.org/T262751 (10jijiki) [06:42:51] ACKNOWLEDGEMENT - Check systemd state on netflow1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. Effie Mouzeli logrotate has failed - T262751 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:42:51] ACKNOWLEDGEMENT - Check systemd state on netflow2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. Effie Mouzeli logrotate has failed - T262751 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:42:51] ACKNOWLEDGEMENT - Check systemd state on netflow3001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. Effie Mouzeli logrotate has failed - T262751 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:42:51] ACKNOWLEDGEMENT - Check systemd state on netflow4001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. Effie Mouzeli logrotate has failed - T262751 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:42:51] ACKNOWLEDGEMENT - Check systemd state on netflow5001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. Effie Mouzeli logrotate has failed - T262751 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200913T0700) [07:55:48] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/definition/{title} (retrieve en-wiktionary definitions for cat) is CRITICAL: Test retrieve en-wiktionary definitions for cat returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [07:57:46] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [11:13:31] 10Operations, 10netops: logrotate failed on all netflow* hosts - https://phabricator.wikimedia.org/T262751 (10Zoranzoki21) [11:14:02] 10Operations, 10netops: every Sunday at 00:00 UTC, logrotate fails on netflow hosts - https://phabricator.wikimedia.org/T257128 (10Zoranzoki21) [14:15:08] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/metadata/{title} (retrieve extended metadata for Video article on English Wikipedia) is CRITICAL: Test retrieve extended metadata for Video article on English Wikipedia returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [14:17:04] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [14:38:31] 10Operations, 10serviceops, 10Patch-For-Review: move 20 new codfw parsoid servers (parse2*) into production - https://phabricator.wikimedia.org/T247441 (10Dzahn) Thank you, i thought i had done 48 hour downtimes on everything but obviously it failed somehow. [15:21:23] (03PS1) 10MarcoAurelio: [itwiki] Increase $wgAutoConfirmAge and $wgAutoConfirmCount [mediawiki-config] - 10https://gerrit.wikimedia.org/r/626876 (https://phabricator.wikimedia.org/T262738) [15:33:36] (03PS1) 10MarcoAurelio: [frwiktionary] Create new namespace "Conjugaison" & associated talk [mediawiki-config] - 10https://gerrit.wikimedia.org/r/626877 (https://phabricator.wikimedia.org/T262298) [16:09:59] thanks for the ACKs, Effie. I was sured i had done 48 hours downtime but obviously it was a fail somehow. [17:00:12] (03PS2) 10ArielGlenn: Improve shell script compatibility, double quoting [puppet] - 10https://gerrit.wikimedia.org/r/624420 (owner: 10Ryan Kemper) [17:00:52] (03CR) 10ArielGlenn: [C: 03+2] Improve shell script compatibility, double quoting [puppet] - 10https://gerrit.wikimedia.org/r/624420 (owner: 10Ryan Kemper) [17:05:11] (03CR) 10ArielGlenn: "I think we only want monitoring of the web service on the primary web server (one of the labstore1006,7 boxes and not the other) even thou" [puppet] - 10https://gerrit.wikimedia.org/r/624328 (owner: 10Dzahn) [17:50:20] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/mobile-sections/{title} (retrieve test page via mobile-sections) is CRITICAL: Test retrieve test page via mobile-sections returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [17:54:14] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [19:44:26] 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, 10RESTBase, and 3 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10cooltey) p:05High→03Unbreak! [19:44:44] 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, 10RESTBase, and 3 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10cooltey) We have received a large number of reports from users that the application shows error code `429`. Do we have any... [19:55:22] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/talk/{title} (Get structured talk page for enwiki Salt article) is CRITICAL: Test Get structured talk page for enwiki Salt article returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [19:57:18] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [20:05:09] 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, 10RESTBase, and 3 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10Dbrant) [21:43:57] 10Operations, 10Wikidata, 10Wikimedia-Mailing-lists: Stop archiving the wikidata-bugs mailinglist in pipermail - https://phabricator.wikimedia.org/T262773 (10Legoktm) [21:44:02] 10Operations, 10Wikimedia-Mailing-lists: Figure out a way to sync old and new mailman - https://phabricator.wikimedia.org/T256539 (10Legoktm) >>! In T256539#6428881, @Legoktm wrote: > Each email has a unique Message-Id (though if a message was cross-posted, it might not be globally unique), which we can use to... [21:49:27] 10Operations, 10Wikidata, 10Wikimedia-Mailing-lists: Stop archiving the wikidata-bugs mailinglist in pipermail - https://phabricator.wikimedia.org/T262773 (10Nemo_bis) At some point, the external archives of wikibugs-l were the only way to search or count certain events in our issue tracker, which were impos... [21:51:36] 10Operations, 10Platform Engineering, 10Product-Infrastructure-Team-Backlog, 10RESTBase, and 3 others: High numbers of HTTP 429 errors - https://phabricator.wikimedia.org/T262691 (10Charlotte) >>! In T262691#6456908, @cooltey wrote: > We have received a large number of reports from users that the applicati... [21:52:38] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/definition/{title} (retrieve en-wiktionary definitions for cat) is CRITICAL: Test retrieve en-wiktionary definitions for cat returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [21:54:34] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [23:00:56] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/mobile-html/{title} (Get page content HTML for test page) is CRITICAL: Test Get page content HTML for test page returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [23:02:56] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [23:47:24] !log Change email address of User:Oversight@enwiki to oversight-l@lists.wikimedia.org as part of OTRS downtime preparation (T262733) [23:47:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:47:32] T262733: Cannot change password for role account that is not an attached global account - Assistance required from someone with shell access - https://phabricator.wikimedia.org/T262733