[00:15:42] PROBLEM - Check systemd state on ms-be2019 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:29:16] RECOVERY - Check systemd state on ms-be2019 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:10:13] musikanimal are you around? [05:23:48] PROBLEM - Check systemd state on wdqs2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [05:36:09] (03PS1) 10ArielGlenn: rebalance number of pages in xml dumps stubs files for large wikis [puppet] - 10https://gerrit.wikimedia.org/r/628570 (https://phabricator.wikimedia.org/T263323) [05:40:58] (03CR) 10ArielGlenn: [C: 03+2] rebalance number of pages in xml dumps stubs files for large wikis [puppet] - 10https://gerrit.wikimedia.org/r/628570 (https://phabricator.wikimedia.org/T263323) (owner: 10ArielGlenn) [06:12:42] RECOVERY - Check systemd state on wdqs2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:18:36] PROBLEM - Check systemd state on wdqs2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:41:52] RECOVERY - Check systemd state on wdqs2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:47:32] PROBLEM - Check systemd state on wdqs2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:53:38] PROBLEM - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is CRITICAL: /v2/suggest/sections/{title}/{from}/{to} (Suggest source sections to translate) is CRITICAL: Test Suggest source sections to translate returned the unexpected status 503 (expecting: 200) https://wikitech.wikimedia.org/wiki/CX [06:55:36] RECOVERY - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/CX [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200920T0700) [07:12:22] RECOVERY - Check systemd state on wdqs2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:18:06] PROBLEM - Check systemd state on wdqs2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:36:11] !log restarting blazegraph + updater on wdqs2002 [07:36:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:36:50] PROBLEM - WDQS high update lag on wdqs2002 is CRITICAL: 1.285e+05 ge 4.32e+04 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [07:36:58] RECOVERY - Query Service HTTP Port on wdqs2002 is OK: HTTP OK: HTTP/1.1 200 OK - 448 bytes in 0.022 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service [07:37:12] RECOVERY - Check systemd state on wdqs2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:42:41] !log depooling wdqs2002 to catch up on lag [07:42:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:43:18] ACKNOWLEDGEMENT - WDQS high update lag on wdqs2002 is CRITICAL: 1.274e+05 ge 4.32e+04 Gehel depooled while catching up on lag https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [08:46:12] !log [urbanecm@mwmaint2001 ~]$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Tepig10102020' 'Davidfromtheworld' # T263317 [08:46:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:46:18] T263317: Unblock stuck global rename of Davidfromtheworld - https://phabricator.wikimedia.org/T263317 [08:49:59] (03CR) 10Urbanecm: [C: 03+1] Enable DiscussionTools beta on metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/627833 (https://phabricator.wikimedia.org/T262984) (owner: 10Esanders) [11:55:10] (03PS1) 10Evrifaessa: Add *.70yearsindonesiaaustralia.com to the wgCopyUploadsDomains allowlist of commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/628581 (https://phabricator.wikimedia.org/T262238) [13:04:53] Hello, I added one patch for deployment but page got broken. https://wikitech.wikimedia.org/wiki/Deployments [13:04:59] Can you fix it, I don't know what to do? [13:06:24] Oh my keyboard.. I wanted to type } but it typed ]... I fixed it now... [13:56:26] (03PS1) 10Evrifaessa: Set 'WT' namespace alias to NS_PROJECT in shn.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/628590 (https://phabricator.wikimedia.org/T256348) [14:11:09] 10Operations, 10Wikidata, 10Wikimedia-Mailing-lists: Stop archiving the wikidata-bugs mailinglist in pipermail - https://phabricator.wikimedia.org/T262773 (10Lydia_Pintscher) 05Open→03Resolved Archiving disabled :) [14:26:31] RECOVERY - WDQS high update lag on wdqs2002 is OK: (C)4.32e+04 ge (W)2.16e+04 ge 2.148e+04 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [14:49:16] 10Operations, 10Wikimedia-Mailing-lists: New version no shows profile pictures - https://phabricator.wikimedia.org/T263355 (10Kizule) [14:49:56] 10Operations, 10Wikimedia-Mailing-lists: New version no shows profile pictures - https://phabricator.wikimedia.org/T263355 (10Kizule) [15:07:09] 10Operations, 10MediaWiki-extensions-Scribunto, 10Wikimedia-production-error: Viewing certain Lua modules on enwiki results in plain-text InvalidArgumentException - https://phabricator.wikimedia.org/T263359 (10AntiCompositeNumber) [15:11:33] 10Operations, 10MediaWiki-extensions-Scribunto, 10Wikimedia-production-error: Viewing certain Lua modules on enwiki results in plain-text InvalidArgumentException - https://phabricator.wikimedia.org/T263359 (10Ammarpad) [15:31:18] 10Operations, 10Wikimedia-Mailing-lists: Hyperkitty in Mailman 3 shows no profile pictures - https://phabricator.wikimedia.org/T263355 (10Aklapper) [15:31:51] 10Operations, 10Wikimedia-Mailing-lists: Hyperkitty in Mailman 3 shows no profile pictures - https://phabricator.wikimedia.org/T263355 (10Aklapper) [17:01:24] (03PS1) 10Evrifaessa: Create Portal and Portal_talk namespaces on trwikisource, and fix an incorrect alias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/628598 (https://phabricator.wikimedia.org/T263358) [17:11:37] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: 102 gt 100 https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [17:15:29] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: (C)100 gt (W)50 gt 36 https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [22:49:15] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [23:04:29] (03PS1) 10Esanders: Simplify lead paragraph check [extensions/MobileFrontend] (wmf/1.36.0-wmf.9) - 10https://gerrit.wikimedia.org/r/628626 [23:14:29] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state