[00:23:09] 08Warning [00:31:19] (03CR) 10Krinkle: [C: 03+1] Enable logging for BlockManager channel at info level [mediawiki-config] - 10https://gerrit.wikimedia.org/r/531299 (https://phabricator.wikimedia.org/T230822) (owner: 10Urbanecm) [00:37:39] RECOVERY - Check systemd state on netbox2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:45:35] PROBLEM - Check systemd state on netbox2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:03:01] 10Operations, 10Performance-Team, 10Traffic, 10media-storage: Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10Krinkle) [02:04:55] 10Operations, 10Performance-Team, 10Traffic, 10media-storage: Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10Krinkle) [02:07:41] 10Operations, 10Traffic, 10media-storage, 10Performance-Team (Radar): Reduce amount of headers sent from web responses - https://phabricator.wikimedia.org/T194814 (10Krinkle) [02:23:10] 08Warning [02:40:02] ÷grumble [03:23:10] 08Warning [05:23:09] 08Warning [06:23:09] 08Warning [06:33:26] PROBLEM - mobileapps endpoints health on scb2006 is CRITICAL: /{domain}/v1/page/summary/{title} (Get summary for test page) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [06:34:30] 10Operations, 10media-storage: outdated DjVu file page thumbnail in cache - https://phabricator.wikimedia.org/T186153 (10Nemo_bis) [06:39:41] RECOVERY - mobileapps endpoints health on scb2006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [06:46:01] PROBLEM - mobileapps endpoints health on scb2006 is CRITICAL: /{domain}/v1/page/media/{title} (Get media in test page) is CRITICAL: Test Get media in test page returned the unexpected status 504 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [06:52:25] RECOVERY - mobileapps endpoints health on scb2006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [07:39:37] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [08:23:09] 08Warning [08:38:09] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:23:10] 08Warning [10:06:09] 10Operations, 10LDAP-Access-Requests: NDA Request from WMDE employee - https://phabricator.wikimedia.org/T231984 (10Peachey88) [10:09:19] 10Operations, 10LDAP-Access-Requests: NDA Request from WMDE employee Raja - https://phabricator.wikimedia.org/T231984 (10Aklapper) [10:35:48] (03PS1) 104nn1l2: Merge branch 'master' of https://gerrit.wikimedia.org/r/operations/mediawiki-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536762 [10:35:50] (03PS1) 104nn1l2: Merge branch 'master' of https://gerrit.wikimedia.org/r/operations/mediawiki-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536763 [10:35:52] (03PS1) 104nn1l2: Add support for some languages on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536764 (https://phabricator.wikimedia.org/T230480) [10:38:32] (03Abandoned) 104nn1l2: Merge branch 'master' of https://gerrit.wikimedia.org/r/operations/mediawiki-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536762 (owner: 104nn1l2) [10:39:19] (03Abandoned) 104nn1l2: Merge branch 'master' of https://gerrit.wikimedia.org/r/operations/mediawiki-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536763 (owner: 104nn1l2) [10:44:52] (03PS2) 104nn1l2: Add support for some languages on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536764 (https://phabricator.wikimedia.org/T230480) [11:23:10] 08Warning [11:37:39] RECOVERY - Check systemd state on netbox2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:40:11] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:45:37] PROBLEM - Check systemd state on netbox2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:07:03] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:21:36] (03CR) 10Zoranzoki21: [C: 04-1] "See comments." (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536764 (https://phabricator.wikimedia.org/T230480) (owner: 104nn1l2) [12:23:10] 08Warning [13:13:59] (03PS3) 104nn1l2: Add support for some languages on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536764 (https://phabricator.wikimedia.org/T230480) [13:21:06] (03CR) 10Zoranzoki21: [C: 03+1] "Should be ok now, @Urbanecm please confirm." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536764 (https://phabricator.wikimedia.org/T230480) (owner: 104nn1l2) [13:38:05] RECOVERY - Check systemd state on netbox2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [13:46:01] PROBLEM - Check systemd state on netbox2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:23:09] 08Warning [14:27:36] Am i missing something or is librenms-wmf message just “warning” [14:35:00] <_joe_> !log test: setting opcache.interned_strings_buffer to 0 on mw1348 for T232613 [14:35:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:35:12] T232613: LBFactoryMulti.php: PHP Notice: Undefined index: - https://phabricator.wikimedia.org/T232613 [15:23:10] 08Warning [15:26:15] (03CR) 10Vgutierrez: [C: 03+1] "NOOP with OCSP enabled: https://puppet-compiler.wmflabs.org/compiler1002/18300/" [puppet] - 10https://gerrit.wikimedia.org/r/536747 (owner: 10Alex Monk) [16:24:43] PROBLEM - Check the last execution of netbox_ganeti_codfw_sync on netbox1001 is CRITICAL: CRITICAL: Status of the systemd unit netbox_ganeti_codfw_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:24:45] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:37:03] RECOVERY - Check systemd state on netbox2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:37:25] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:44:59] PROBLEM - Check systemd state on netbox2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:45:53] RECOVERY - Check the last execution of netbox_ganeti_codfw_sync on netbox1001 is OK: OK: Status of the systemd unit netbox_ganeti_codfw_sync https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:49:16] 10Operations, 10Cloud-Services, 10SRE-Access-Requests, 10Developer-Advocacy (Jul-Sep 2019): Membership in "researchers" group for Srishti Sethi - https://phabricator.wikimedia.org/T232664 (10srishakatux) Script is here T226663#5287195 [16:51:23] !log Fixed a dozen abuse filtes, listed at https://phabricator.wikimedia.org/T156096#5494060. The trailing pipe character was removed from filters that had it which is no longer supported in a future version of AbuseFilter. [16:51:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:04:03] (03PS6) 10Andrew Bogott: codfw1dev: First pass at building out cloudweb2001-dev [puppet] - 10https://gerrit.wikimedia.org/r/536672 (https://phabricator.wikimedia.org/T229441) [17:06:43] (03CR) 10Andrew Bogott: [C: 03+2] codfw1dev: First pass at building out cloudweb2001-dev [puppet] - 10https://gerrit.wikimedia.org/r/536672 (https://phabricator.wikimedia.org/T229441) (owner: 10Andrew Bogott) [17:16:48] PROBLEM - Check systemd state on cloudweb2001-dev is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:19:18] PROBLEM - Memory correctable errors -EDAC- on elastic1029 is CRITICAL: 4.001 ge 4 https://wikitech.wikimedia.org/wiki/Monitoring/Memory%23Memory_correctable_errors_-EDAC- https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=elastic1029&var-datasource=eqiad+prometheus/ops [17:20:14] PROBLEM - HHVM rendering on mw2290 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [17:20:14] ACKNOWLEDGEMENT - Check systemd state on cloudweb2001-dev is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. andrew bogott These were silenced until a second ago. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:21:08] RECOVERY - HHVM rendering on mw2290 is OK: HTTP OK: HTTP/1.1 200 OK - 75820 bytes in 0.327 second response time https://wikitech.wikimedia.org/wiki/Application_servers [17:23:09] 08Warning [17:42:54] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:50:48] PROBLEM - SSH wtp1031.mgmt on wtp1031.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [18:23:10] 08Warning [18:37:36] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:10:01] Hi, CI no runs for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/BlueSpiceFoundation/+/536735/ [19:10:04] What happening? [19:11:07] Doesn't look like they're whitelisted [19:11:24] I am whitelisted [19:11:27] And tests no run [19:12:04] On zuul nothing for 536735 https://integration.wikimedia.org/zuul/ [19:12:22] In postmerge are two mediawiki/core patches [19:19:52] paladox: No, recheck no works. I saw your comment [19:20:20] Yes, because the repo is likley not defined in integration/config (haven't checked but based on the behavour that may be why) [19:20:55] How if it runs gate-and-submit previously? [19:21:24] https://github.com/wikimedia/integration-config/blob/26fb84df00826c6cd2870b5b404e34457704906a/zuul/layout.yaml#L5384 [19:21:34] PROBLEM - Memcached on cloudweb2001-dev is CRITICAL: connect to address 208.80.153.60 and port 11000: Connection refused https://wikitech.wikimedia.org/wiki/Memcached [19:22:11] Yes, I saw it is defined [19:22:37] Oh heh [19:23:01] extension-quibble-composer-nohhvm [19:23:08] It runs jobs which are not hhvm? [19:30:08] RECOVERY - Check systemd state on cloudweb2001-dev is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:34:54] PROBLEM - Check systemd state on cloudweb2001-dev is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:39:28] PROBLEM - mediawiki-installation DSH group on cloudweb2001-dev is CRITICAL: Host cloudweb2001-dev is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [20:09:09] (03PS2) 10Gergő Tisza: Update ORES filter threshold configuration for new huwiki model [mediawiki-config] - 10https://gerrit.wikimedia.org/r/536732 (https://phabricator.wikimedia.org/T230031) [20:23:10] 08Warning [20:43:30] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [20:46:52] Anyone to help me with this: https://phabricator.wikimedia.org/T232877 [20:47:05] stashbot: T232877 [20:47:06] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [20:47:06] T232877: The "RevisionInsertComplete" hook is listed in documentation as deprecated but doesn't emit deprecation warnings yet - https://phabricator.wikimedia.org/T232877 [21:00:42] 10Operations, 10Wikimedia-Mailing-lists, 10Wikispore: Wikispore mailing list - https://phabricator.wikimedia.org/T232961 (10Pharos) [21:11:08] RECOVERY - Check systemd state on cloudweb2001-dev is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [21:15:50] PROBLEM - Check systemd state on cloudweb2001-dev is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [21:23:09] 08Warning [21:37:08] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [22:02:13] 10Operations, 10Wikimedia-Mailing-lists, 10Wikispore: Wikispore mailing list - https://phabricator.wikimedia.org/T232961 (10Tgr) Personally, I'd go with Wikimedia Space instead... [22:14:46] 10Operations, 10Wikimedia-Mailing-lists, 10Wikispore: Wikispore mailing list - https://phabricator.wikimedia.org/T232961 (10Pharos) While Wikimedia Space might be nice too as a supplement, it's still a niche platform, and the vast majority of potential Wikimedians interested in this project do use email. I... [22:53:36] RECOVERY - SSH wtp1031.mgmt on wtp1031.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [23:23:09] 08Warning [23:39:48] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [23:56:25] oh you