[00:20:17] PROBLEM - IPv6 ping to eqsin on ripe-atlas-eqsin IPv6 is CRITICAL: CRITICAL - failed 36 probes of 393 (alerts on 35) - https://atlas.ripe.net/measurements/11645088/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts [00:25:29] RECOVERY - IPv6 ping to eqsin on ripe-atlas-eqsin IPv6 is OK: OK - failed 19 probes of 393 (alerts on 35) - https://atlas.ripe.net/measurements/11645088/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts [01:03:39] PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb={LIST,PATCH,PUT} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:03:57] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb={LIST,PUT} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:03:59] PROBLEM - etcd request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 operation={compareAndSwap,get,list} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:25] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation={compareAndSwap,get,list} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:47] PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb={PATCH,PUT} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:47] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation={get,list} https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:09] RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:09] RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:13] RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:31] RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:35] RECOVERY - etcd request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:57] RECOVERY - etcd request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:37:01] PROBLEM - Apache HTTP on mw1281 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.096 second response time [02:37:15] PROBLEM - Nginx local proxy to apache on mw1281 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.152 second response time [02:38:19] RECOVERY - Apache HTTP on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 2.425 second response time [02:38:29] RECOVERY - Nginx local proxy to apache on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.192 second response time [02:52:55] (03CR) 10Krinkle: "I don't know anything about mcrouter, but one thing to keep in mind here is that Aaron's change only affects the mw-wan route, not the reg" [puppet] - 10https://gerrit.wikimedia.org/r/492948 (owner: 10Aaron Schulz) [03:37:43] PROBLEM - puppet last run on mw2234 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPCity.dat.gz] [04:03:57] RECOVERY - puppet last run on mw2234 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [05:04:42] 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for sukhe - https://phabricator.wikimedia.org/T217438 (10Tbayer) See also https://wikitech.wikimedia.org/wiki/Analytics/Data_access [08:53:29] 10Operations, 10Wikimedia-Mailing-lists: Close the grwp-wici mailing list - https://phabricator.wikimedia.org/T217247 (10Llywelyn2000) Yes, thanks! [08:54:03] 10Operations, 10Wikimedia-Mailing-lists: Close the grwp-wici mailing list - https://phabricator.wikimedia.org/T217247 (10Llywelyn2000) Can this list be wholly removed? Yes, thanks! [09:58:39] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:44] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 5.035 second response time [10:11:19] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:48] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 2.739 second response time [10:18:19] (03PS1) 10Ammarpad: Add editcontentmodel right to the templateeditor group on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494016 (https://phabricator.wikimedia.org/T217499) [10:21:03] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:44:15] !log restart pdfrender on scb1003 [10:44:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:44:59] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.079 second response time [11:25:18] (03CR) 10D3r1ck01: Add editcontentmodel right to the templateeditor group on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494016 (https://phabricator.wikimedia.org/T217499) (owner: 10Ammarpad) [11:52:05] (03CR) 10Framawiki: [C: 04-1] Added and subdomains of mehrnews.com to wgCopyUploadDomains (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492448 (https://phabricator.wikimedia.org/T213961) (owner: 10Zoranzoki21) [12:26:22] !log restarted icinga on icinga2001, stale status file, too many open files [12:26:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:31:41] (03PS2) 10Ammarpad: Add editcontentmodel right to the templateeditor group on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494016 (https://phabricator.wikimedia.org/T217499) [12:33:32] (03CR) 10Ammarpad: Add editcontentmodel right to the templateeditor group on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494016 (https://phabricator.wikimedia.org/T217499) (owner: 10Ammarpad) [12:33:34] (03CR) 10D3r1ck01: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494016 (https://phabricator.wikimedia.org/T217499) (owner: 10Ammarpad) [12:34:07] (03CR) 10MarcoAurelio: Add editcontentmodel right to the templateeditor group on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494016 (https://phabricator.wikimedia.org/T217499) (owner: 10Ammarpad) [12:35:59] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.073 second response time [15:58:54] (03PS1) 10Krinkle: logstash: Remove filter for unused 'exception-json' channel [puppet] - 10https://gerrit.wikimedia.org/r/494042 (https://phabricator.wikimedia.org/T136849) [15:59:43] (03PS2) 10Krinkle: logstash: Remove filter for unused 'exception-json' channel [puppet] - 10https://gerrit.wikimedia.org/r/494042 (https://phabricator.wikimedia.org/T136849) [16:06:08] (03PS3) 10Zoranzoki21: wgCopyUploadDomains: Changed domain for mehrnews.com [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492448 (https://phabricator.wikimedia.org/T213961) [19:26:49] (03Abandoned) 10Paladox: php: Add support for puppet6 [puppet] - 10https://gerrit.wikimedia.org/r/481269 (owner: 10Paladox) [19:26:53] (03Abandoned) 10Paladox: wmlib: Fix support for puppet6 in php_ini.rb, ini.rb and ordered_yaml.rb [puppet] - 10https://gerrit.wikimedia.org/r/481254 (owner: 10Paladox) [19:26:59] (03Abandoned) 10Paladox: wmflib: Add support for puppet6 in require_package [puppet] - 10https://gerrit.wikimedia.org/r/481271 (owner: 10Paladox) [20:31:43] (03CR) 10Gergő Tisza: [C: 03+1] logstash: Remove filter for unused 'exception-json' channel [puppet] - 10https://gerrit.wikimedia.org/r/494042 (https://phabricator.wikimedia.org/T136849) (owner: 10Krinkle) [20:49:46] 10Operations, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review, and 3 others: Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10aaron) getWithSetCallback() should not internally... [21:32:32] (03CR) 10Framawiki: [bugfix] disable crosswiki upload till a solution is found for the broken images (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491928 (https://phabricator.wikimedia.org/T214230) (owner: 10Mahveotm) [22:10:11] (03PS2) 10Mahveotm: [bugfix] disable crosswiki upload till a solution is found for the broken images [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491928 (https://phabricator.wikimedia.org/T214230) [22:14:40] (03PS3) 10Mahveotm: [bugfix] disable crosswiki upload till a solution is found for the broken images [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491928 (https://phabricator.wikimedia.org/T214230) [22:19:20] (03CR) 10Mahveotm: [bugfix] disable crosswiki upload till a solution is found for the broken images (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491928 (https://phabricator.wikimedia.org/T214230) (owner: 10Mahveotm)