[00:24:19] PROBLEM - Number of backend failures per minute from CirrusSearch on graphite1004 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [600.0] https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=1&var-cluster=eqiad&var-smoothing=1&panelId=9&fullscreen [00:29:07] RECOVERY - Number of backend failures per minute from CirrusSearch on graphite1004 is OK: OK: Less than 20.00% above the threshold [300.0] https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=1&var-cluster=eqiad&var-smoothing=1&panelId=9&fullscreen [00:39:17] 10Operations, 10WMF-Legal, 10serviceops: Move old transparency report pages to historical URLs - https://phabricator.wikimedia.org/T230638 (10Varnent) [00:39:41] 10Operations, 10WMF-Legal, 10serviceops: Move old transparency report pages to historical URLs and setup redirect - https://phabricator.wikimedia.org/T230638 (10Varnent) [01:29:42] 10Operations, 10Wikimedia-Mailing-lists: Set up mailing list for Santali Wikipedia - https://phabricator.wikimedia.org/T230435 (10Manik87) @CDanis Thak you very much for your prompt action. Have a good day! [02:38:11] PROBLEM - Postgres Replication Lag on maps2001 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 39663088 and 2 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [02:39:43] PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 153406496 and 7 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [02:47:53] RECOVERY - Postgres Replication Lag on maps2001 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 338344 and 61 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [02:49:23] RECOVERY - Postgres Replication Lag on maps1002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 831208 and 88 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [03:31:45] RECOVERY - Logstash rate of ingestion percent change compared to yesterday on icinga1001 is OK: (C)130 ge (W)110 ge 106.8 https://phabricator.wikimedia.org/T202307 https://grafana.wikimedia.org/dashboard/db/logstash?orgId=1&panelId=2&fullscreen [08:25:37] https://upload.wikimedia.org/wikipedia/commons/thumb/3/32/Die_Sch%C3%B6llenen_Schlucht_mit_Teufelsbr%C3%BCcke_im_schweizerischen_Kanton_Uri.jpg/2500px-Die_Sch%C3%B6llenen_Schlucht_mit_Teufelsbr%C3%BCcke_im_schweizerischen_Kanton_Uri.jpg [08:25:50] Request from 2401:4900:16d5:75ed:f021:9d5f:65c3:861f via cp5004 frontend, Varnish XID 680386932 [08:25:51] Upstream caches: cp5004 int [08:25:51] Error: 500, Internal Server Error at Sat, 17 Aug 2019 08:25:15 GMT [10:23:02] (03CR) 10Reedy: [C: 03+1] Send 33.3% of anonymous users to PHP7.2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/529924 (https://phabricator.wikimedia.org/T219150) (owner: 10Effie Mouzeli) [11:41:35] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/page/summary/{title} (Get summary for test page) timed out before a response was received https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [11:43:05] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29 [11:58:21] PROBLEM - Nginx local proxy to apache on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [11:59:49] RECOVERY - Nginx local proxy to apache on mw1235 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.045 second response time https://wikitech.wikimedia.org/wiki/Application_servers [12:04:51] (03PS9) 10Ladsgroup: mediawiki: Use mediawiki::errorpage instead of a hhvm-fatal-error.php.erb [puppet] - 10https://gerrit.wikimedia.org/r/511078 (https://phabricator.wikimedia.org/T113114) [15:37:07] PROBLEM - Check the Netbox report-s- puppetdb for fail status. on netmon1002 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports [15:46:25] (03CR) 10Krinkle: mediawiki: Use mediawiki::errorpage instead of a hhvm-fatal-error.php.erb (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/511078 (https://phabricator.wikimedia.org/T113114) (owner: 10Ladsgroup) [19:01:07] 10Operations, 10DNS, 10Traffic, 10Wikimedia-Apache-configuration, 10Patch-For-Review: Remove aliases `minnan` and `zh-cfr` for `nan`/`zh-min-nan` - https://phabricator.wikimedia.org/T230382 (10Aklapper) [19:34:46] 10Operations, 10DNS, 10Traffic, 10Wikimedia-Apache-configuration, 10Patch-For-Review: Remove aliases `minnan` and `zh-cfr` for the Min Nan Wikipedia - https://phabricator.wikimedia.org/T230382 (10Fomafix) [21:25:12] (03PS1) 10Alexandros Kosiaris: mediawiki:errorpage: Make content default undef [puppet] - 10https://gerrit.wikimedia.org/r/530712 (https://phabricator.wikimedia.org/T113114) [21:27:17] (03CR) 10Alexandros Kosiaris: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/530712 (https://phabricator.wikimedia.org/T113114) (owner: 10Alexandros Kosiaris) [21:37:54] (03PS2) 10Alexandros Kosiaris: mediawiki:errorpage: Make content default undef [puppet] - 10https://gerrit.wikimedia.org/r/530712 (https://phabricator.wikimedia.org/T113114) [21:45:02] (03PS3) 10Alexandros Kosiaris: mediawiki:errorpage: Make content default undef [puppet] - 10https://gerrit.wikimedia.org/r/530712 (https://phabricator.wikimedia.org/T113114) [21:45:34] (03CR) 10jerkins-bot: [V: 04-1] mediawiki:errorpage: Make content default undef [puppet] - 10https://gerrit.wikimedia.org/r/530712 (https://phabricator.wikimedia.org/T113114) (owner: 10Alexandros Kosiaris) [21:53:23] (03PS4) 10Alexandros Kosiaris: mediawiki:errorpage: Make content default undef [puppet] - 10https://gerrit.wikimedia.org/r/530712 (https://phabricator.wikimedia.org/T113114) [21:59:18] (03CR) 10Alexandros Kosiaris: "Should be rebased on top of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/530712/" [puppet] - 10https://gerrit.wikimedia.org/r/511078 (https://phabricator.wikimedia.org/T113114) (owner: 10Ladsgroup)