[00:14:44] Q; Someone (not a bot) editing at 1000 edits/minute via automated tools — would you consider it normal? Or something that needs to be pulled over? [00:55:29] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 34.83 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [00:55:54] PROBLEM - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:56:14] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:56:22] PROBLEM - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:56:28] o/ [00:56:32] PROBLEM - LVS HTTP IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:56:37] I get timeouts trying to edit on mediawiki.org. [00:57:05] here [00:57:06] PROBLEM - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:57:40] RECOVERY - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 1.025 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:57:51] PROBLEM - Restbase edge esams on text-lb.esams.wikimedia.org is CRITICAL: /api/rest_v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received: /api/rest_v1/page/media-list/{title} (Get media-list from storage) timed out before a response was received: /api/rest_v1/page/mobile-html/{title} (Get mobile-html from storage) timed out before a response was received: /api/rest_v [00:57:51] k/{type} (Mathoid - check test formula) timed out before a response was received: /api/rest_v1/page/summary/{title} (Get summary from storage) timed out before a response was received: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before a response was received: /api/rest_v1/transform/wikitext/to/html/{title} (Transform wikitext to html) timed out before a response was received: /api/rest_v1/page/mobil [00:57:51] } (Get mobile-sections for a test page on enwiki) timed out before a response was received: /api/rest_v1/page/html/{title} (Get html by title from storage) timed out before a response was received: /api/rest_v1/page/references/{title} (Get references from storage) timed out before a response was received: /api/rest_v1/feed/announcements (Retrieve announcements) timed out before a response was received: /api/rest_v1/feed/featured/ [00:57:51] (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase [00:58:00] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.511 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:58:08] RECOVERY - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 0.344 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:58:18] RECOVERY - LVS HTTP IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 551 bytes in 0.167 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:58:50] RECOVERY - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15184 bytes in 0.501 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [00:59:35] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [01:02:57] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 87.2 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:02:59] andre__: we should be back to normal, are you still having any trouble? [01:03:07] rlazarus, WFM. Thanks [01:03:39] thanks for the report! [01:09:12] (03CR) 10CDanis: [C: 03+2] GeoDNS: Define alternate esams depooling method [dns] - 10https://gerrit.wikimedia.org/r/565049 (owner: 10BBlack) [01:09:24] (03PS2) 10CDanis: GeoDNS: Define alternate esams depooling method [dns] - 10https://gerrit.wikimedia.org/r/565049 (owner: 10BBlack) [01:10:49] (03PS1) 10CDanis: depool esams and reroute US to codfw [dns] - 10https://gerrit.wikimedia.org/r/567298 [01:11:28] (03CR) 10CDanis: [C: 03+2] depool esams and reroute US to codfw [dns] - 10https://gerrit.wikimedia.org/r/567298 (owner: 10CDanis) [01:12:06] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 46.7 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:12:16] !log depool esams with new geo-maps-esams-offline [01:12:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:12:50] !log deployed [01:12:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:15:37] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 82.27 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:19:13] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 52.86 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:24:39] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:33:45] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 10.05 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:35:33] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:40:53] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is CRITICAL: 59.93 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:47:14] those last ones are expected [01:51:41] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is OK: (C)60 le (W)70 le 72.62 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [02:01:02] (03PS1) 10CDanis: Revert "depool esams and reroute US to codfw" [dns] - 10https://gerrit.wikimedia.org/r/567300 [04:45:01] (03PS1) 10Ammarpad: Add Commons to enwiki importsources [mediawiki-config] - 10https://gerrit.wikimedia.org/r/567305 (https://phabricator.wikimedia.org/T242884) [04:47:15] (03PS1) 10Ammarpad: Enable new user message for auto-created accounts on zh_classical wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/567306 (https://phabricator.wikimedia.org/T243509) [05:05:14] (03PS1) 10Ammarpad: Update logo for zh_classical wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/567307 (https://phabricator.wikimedia.org/T243509) [08:48:49] PROBLEM - Maps tiles generation on icinga1001 is CRITICAL: CRITICAL: 90.42% of data under the critical threshold [5.0] https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1 [10:36:14] (03CR) 10Effie Mouzeli: [C: 03+2] Revert "depool esams and reroute US to codfw" [dns] - 10https://gerrit.wikimedia.org/r/567300 (owner: 10CDanis) [10:37:02] !log Pool esams back [10:37:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:38:56] !log deployed [10:38:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:49:07] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is CRITICAL: 43.86 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [11:21:37] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [11:23:49] !log restarted tilerator and tileratorui on maps1001 [11:23:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:25:03] !log restarted tilerator and tileratorui on maps1002 [11:25:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:40:17] PROBLEM - Host elastic2043 is DOWN: PING CRITICAL - Packet loss = 100% [11:42:01] RECOVERY - Host elastic2043 is UP: PING OK - Packet loss = 0%, RTA = 139.93 ms [14:35:01] PROBLEM - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:35:25] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 23.7 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [14:36:33] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:36:35] Hi getting unusual timeouts again... Anyone know where to get an analysis tool [14:36:44] that gives the team the information they need? [14:36:55] RECOVERY - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 8.018 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:37:07] Hello, I can't open Phabricator. What's happening? [14:37:39] wikidata down [14:37:47] nowiki down [14:37:49] enwiki loads with errors [14:37:50] enwiki down [14:37:52] Oh, I can't open srwiki [14:37:55] That is in Norway [14:38:20] dewiki slow/down [14:38:24] Netherlands too [14:38:34] Serbia too [14:38:45] I can open just Gerrit :P [14:38:58] even grafana is very slow [14:39:03] or down [14:39:22] * Urbanecm can load WMF projects only on US-based VPN [14:39:31] PROBLEM - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is CRITICAL: /api/rest_v1/transform/wikitext/to/html/{title} (Transform wikitext to html) timed out before a response was received: /api/rest_v1/media/math/check/{type} (Mathoid - check test formula) timed out before a response was received: /api/rest_v1/page/summary/{title} (Get summary from storage) timed out before a response was received: /api/rest_v1/feed/featured/ [14:39:31] (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /api/rest_v1/page/media-list/{title} (Get media-list from storage) timed out before a response was received: /api/rest_v1/page/references/{title} (Get references from storage) timed out before a response was received: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before a response was received https://wikite [14:39:31] wiki/RESTBase [14:39:35] Very slow here [14:39:37] UK [14:39:49] PROBLEM - LVS HTTP IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:39:57] PROBLEM - graphoid endpoints health on scb1003 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:39:57] we're aware and looking into it. thanks for the reports. [14:40:09] PROBLEM - LVS HTTP IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:40:09] PROBLEM - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Graphoid [14:40:35] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.20.0.15: Connection reset by peer https://wikitech.wikimedia.org/wiki/PyBal [14:40:58] thanks apergos [14:41:01] PROBLEM - graphoid endpoints health on scb1001 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:41:26] Thanks arpegos [14:41:27] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received https://wikitech.wikimedia.org/wiki/Citoid [14:41:35] RECOVERY - LVS HTTP IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 159 bytes in 2.532 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:41:46] Thanks apergos [14:41:57] RECOVERY - LVS HTTP IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 159 bytes in 4.542 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:42:11] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.724 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:42:21] PROBLEM - proton endpoints health on proton1002 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/pro [14:42:25] RECOVERY - PyBal backends health check on lvs3005 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [14:42:38] see https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&fullscreen&panelId=6 [14:42:55] RECOVERY - graphoid endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:43:11] RECOVERY - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [14:43:23] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Citoid [14:46:41] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 109.7 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [14:47:49] RECOVERY - proton endpoints health on proton1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [14:48:27] PROBLEM - graphoid endpoints health on scb1001 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:48:55] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received https://wikitech.wikimedia.org/wiki/Citoid [14:48:59] PROBLEM - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is CRITICAL: /api/rest_v1/page/mobile-sections/{title} (Get mobile-sections for a test page on enwiki) timed out before a response was received: /api/rest_v1/page/talk/{title} (Get structured talk page for enwiki Salt article) timed out before a response was received: /api/rest_v1/feed/announcements (Retrieve announcements) timed out before a response was received: /api [14:48:59] th/check/{type} (Mathoid - check test formula) timed out before a response was received: /api/rest_v1/page/html/{title} (Get html by title from storage) timed out before a response was received: /api/rest_v1/page/mobile-html/{title} (Get mobile-html from storage) timed out before a response was received: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before a response was received: /api/rest_v1/transfor [14:48:59] l/{title} (Transform wikitext to html) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase [14:49:05] PROBLEM - proton endpoints health on proton1001 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/pro [14:49:13] PROBLEM - graphoid endpoints health on scb1004 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:49:25] RECOVERY - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Graphoid [14:50:03] PROBLEM - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:50:17] RECOVERY - graphoid endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:50:23] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 18.2 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [14:51:35] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:52:21] PROBLEM - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:52:27] RECOVERY - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [14:52:39] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Citoid [14:52:49] RECOVERY - proton endpoints health on proton1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [14:53:07] PROBLEM - graphoid endpoints health on scb1002 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:53:43] RECOVERY - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 3.692 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:54:26] PROBLEM - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:54:45] RECOVERY - graphoid endpoints health on scb1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:54:50] I'm not seeing any long delays in a traceroute [14:54:51] RECOVERY - graphoid endpoints health on scb1003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:54:51] RECOVERY - graphoid endpoints health on scb1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [14:54:55] hi, i'm seeing massive IPv6 packet loss to de.wikipedia.org and meta.wikipedia.org. is this known? packets are lost at last hop, text-lb.esams.wikimedia.org - i'm accessing from AS8881 [14:55:07] PROBLEM - LVS HTTP IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:55:39] tomreyn: yes we are working on it :( [14:55:39] for 91.198.174.192, I'm seeing ping times of about 20s but no content comes back [14:55:48] thanks! [14:55:56] thanks for the reports [14:55:56] So I don't think this is a network issue my end... [14:56:01] RECOVERY - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 1.078 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:56:11] 19-22ms sorry [14:56:14] !log enabling netflow sampling on the knams-esams links (esams side) [14:56:15] RECOVERY - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15183 bytes in 3.173 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:56:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:56:53] RECOVERY - LVS HTTP IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 550 bytes in 0.167 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:57:09] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.516 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [14:57:40] Hey guys, why is this happenning 3 times a day every last 3 days? Is this some continuous attack effort or some continous issues with servers, [14:57:50] ? [14:58:22] Dvorapa: It's under investigation, I don't think the relevant people can comment yet [14:58:27] I see [14:58:35] Hopefully just a small issue [14:59:06] Has there been any controversoy on Wikipedia recently? [14:59:17] *controversy? [14:59:26] (03PS1) 10CDanis: Revert "Revert "depool esams and reroute US to codfw"" [dns] - 10https://gerrit.wikimedia.org/r/567343 [14:59:45] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 86.33 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [14:59:51] (03CR) 10BBlack: [C: 03+1] Revert "Revert "depool esams and reroute US to codfw"" [dns] - 10https://gerrit.wikimedia.org/r/567343 (owner: 10CDanis) [15:00:09] (03CR) 10CDanis: [C: 03+2] Revert "Revert "depool esams and reroute US to codfw"" [dns] - 10https://gerrit.wikimedia.org/r/567343 (owner: 10CDanis) [15:00:30] !log depool esams [15:00:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:00:52] what happened to esams networking there? traffic spike? [15:01:01] !log deployed [15:01:01] we are investigating and taking measures [15:01:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:01:05] cool [15:01:05] more updates when we have them [15:01:09] i'd say let them handle the immediate incident first, they'll have their hands full right now. [15:01:52] thanks :) [15:07:12] Still gets timeouts at wikidata [15:07:58] no updates yet, please be patient [15:09:51] It's coming back slowly for me, but can be a bit erratic [15:09:53] PROBLEM - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Graphoid [15:10:03] PROBLEM - proton endpoints health on proton1002 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [15:10:23] PROBLEM - restbase endpoints health on restbase1020 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:10:25] PROBLEM - restbase endpoints health on restbase1021 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:10:25] PROBLEM - restbase endpoints health on restbase1025 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:10:27] PROBLEM - https://phabricator.wikimedia.org #page on phabricator.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Phabricator [15:10:29] PROBLEM - restbase endpoints health on restbase1026 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:10:29] PROBLEM - restbase endpoints health on restbase1022 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:10:31] PROBLEM - restbase endpoints health on restbase1023 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:10:43] PROBLEM - graphoid endpoints health on scb1001 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:10:49] PROBLEM - LVS HTTP IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:11:11] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received https://wikitech.wikimedia.org/wiki/Citoid [15:11:25] PROBLEM - proton endpoints health on proton1001 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/pro [15:11:29] PROBLEM - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is CRITICAL: WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by ReadTimeoutError(HTTPSConnectionPool(host=text-lb.eqiad.wikimedia.org, port=443): Read timed out. (read timeout=15),): /api/rest_v1/?spec https://wikitech.wikimedia.org/wiki/RESTBase [15:11:33] PROBLEM - graphoid endpoints health on scb1002 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:11:35] PROBLEM - graphoid endpoints health on scb1003 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:11:37] PROBLEM - restbase endpoints health on restbase1027 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:11:47] At least I know get the WMF error [15:11:49] *now [15:11:51] :) [15:12:11] RECOVERY - restbase endpoints health on restbase1020 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:12:17] RECOVERY - restbase endpoints health on restbase1021 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:12:17] RECOVERY - restbase endpoints health on restbase1025 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:12:19] RECOVERY - restbase endpoints health on restbase1026 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:12:19] RECOVERY - restbase endpoints health on restbase1023 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:12:19] RECOVERY - restbase endpoints health on restbase1022 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:12:30] "Request from 88.97.96.89 via cp3050 frontend, Varnish XID 506671518 [15:12:30] Error: 503, Backend fetch failed at Sun, 26 Jan 2020 15:11:26 GMT" [15:12:35] RECOVERY - graphoid endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:12:45] PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:12:57] PROBLEM - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:13:01] PROBLEM - restbase endpoints health on restbase-dev1005 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:13:35] PROBLEM - LVS HTTP IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:13:37] PROBLEM - LVS HTTP IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:13:37] RECOVERY - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Graphoid [15:13:55] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:14:15] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - testlb_443: Servers cp3060.esams.wmnet, cp3062.esams.wmnet, cp3064.esams.wmnet, cp3058.esams.wmnet, cp3052.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb_443: Servers cp3050.esams.wmnet, cp3062.esams.wmnet, cp3058.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: testlb6_443: Servers cp3060.esams.wmnet, cp3058.e [15:14:15] 2.esams.wmnet, cp3064.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb6_443: Servers cp3064.esams.wmnet, cp3058.esams.wmnet, cp3052.esams.wmnet, cp3056.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [15:14:19] PROBLEM - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:14:39] PROBLEM - Restbase edge esams on text-lb.esams.wikimedia.org is CRITICAL: /api/rest_v1/page/summary/{title} (Get summary from storage) timed out before a response was received: /api/rest_v1/page/html/{title} (Get html by title from storage) timed out before a response was received: /api/rest_v1/transform/wikitext/to/html/{title} (Transform wikitext to html) timed out before a response was received: /api/rest_v1/page/talk/{title} [15:14:39] alk page for enwiki Salt article) timed out before a response was received: /api/rest_v1/page/mobile-sections/{title} (Get mobile-sections for a test page on enwiki) timed out before a response was received: /api/rest_v1/page/references/{title} (Get references from storage) timed out before a response was received: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before a response was received: /api/rest_ [15:14:39] ents (Retrieve announcements) timed out before a response was received: /api/rest_v1/page/media-list/{title} (Get media-list from storage) timed out before a response was received: /api/rest_v1/page/mobile-html/{title} (Get mobile-html from storage) timed out before a response was received: /api/rest_v1/media/math/check/{type} (Mathoid - check test formula) timed out before a response was received https://wikitech.wikimedia.org/w [15:14:41] RECOVERY - LVS HTTP IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 563 bytes in 7.368 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:14:47] PROBLEM - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:14:59] RECOVERY - restbase endpoints health on restbase-dev1005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:15:27] RECOVERY - restbase endpoints health on restbase1027 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:15:37] PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:15:47] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 1.846 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:16:19] RECOVERY - https://phabricator.wikimedia.org #page on phabricator.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 36942 bytes in 7.652 second response time https://wikitech.wikimedia.org/wiki/Phabricator [15:16:19] PROBLEM - PyBal IPVS diff check on lvs3005 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.20.0.15: Connection reset by peer https://wikitech.wikimedia.org/wiki/PyBal [15:16:35] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 1.339 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [15:16:37] RECOVERY - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 0.753 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:16:55] RECOVERY - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15183 bytes in 3.839 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:17:15] PROBLEM - mobileapps endpoints health on scb1001 is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:18:09] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - testlb_443: Servers cp3062.esams.wmnet, cp3052.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb_443: Servers cp3060.esams.wmnet, cp3054.esams.wmnet, cp3062.esams.wmnet, cp3064.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: testlb6_443: Servers cp3050.esams.wmnet, cp3054.esams.wmnet, cp3058.esams.wmnet, cp3062.e [15:18:09] 6.esams.wmnet are marked down but pooled: textlb6_443: Servers cp3060.esams.wmnet, cp3050.esams.wmnet, cp3054.esams.wmnet, cp3062.esams.wmnet, cp3056.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [15:18:35] PROBLEM - graphoid endpoints health on scb1001 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:19:03] PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:19:21] RECOVERY - LVS HTTP IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 550 bytes in 1.806 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:19:23] RECOVERY - LVS HTTP IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 159 bytes in 3.319 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:19:37] PROBLEM - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Graphoid [15:20:21] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [15:20:21] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [15:20:25] PROBLEM - restbase endpoints health on restbase-dev1006 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:20:41] PROBLEM - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:20:51] RECOVERY - mobileapps endpoints health on scb1003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:20:53] RECOVERY - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [15:20:55] RECOVERY - mobileapps endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:20:59] RECOVERY - proton endpoints health on proton1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [15:21:09] RECOVERY - graphoid endpoints health on scb1003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:21:11] RECOVERY - graphoid endpoints health on scb1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:21:17] RECOVERY - mobileapps endpoints health on scb1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:21:23] RECOVERY - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Graphoid [15:21:25] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by ReadTimeoutError(HTTPSConnectionPool(host=text-lb.esams.wikimedia.org, port=443): Read timed out.,): /api/rest_v1/?spec https://wikitech.wikimedia.org/wiki/RESTBase [15:21:26] (03PS1) 10Elukey: Revert "Revert "Revert "depool esams and reroute US to codfw""" [dns] - 10https://gerrit.wikimedia.org/r/567344 [15:21:37] RECOVERY - proton endpoints health on proton1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [15:22:11] RECOVERY - restbase endpoints health on restbase-dev1006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [15:22:15] RECOVERY - graphoid endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:22:21] RECOVERY - mobileapps endpoints health on scb1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [15:22:24] RECOVERY - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.075 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:22:39] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Citoid [15:23:49] PROBLEM - PyBal connections to etcd on lvs3005 is CRITICAL: CRITICAL: 0 connections established with conf1006.eqiad.wmnet:4001 (min=12) https://wikitech.wikimedia.org/wiki/PyBal [15:24:05] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [15:24:14] (03CR) 10Elukey: [C: 03+2] Revert "Revert "Revert "depool esams and reroute US to codfw""" [dns] - 10https://gerrit.wikimedia.org/r/567344 (owner: 10Elukey) [15:24:23] !log repool esams [15:24:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:25:33] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - testlb_443: Servers cp3062.esams.wmnet, cp3052.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb_443: Servers cp3060.esams.wmnet, cp3050.esams.wmnet, cp3064.esams.wmnet are marked down but pooled: testlb6_443: Servers cp3062.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: ncredirlb_443: Servers ncredir3002.esams. [15:25:33] down but pooled: textlb_80: Servers cp3052.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [15:25:39] RECOVERY - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 5.360 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:26:41] !log repool deployed [15:26:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:27:25] RECOVERY - PyBal IPVS diff check on lvs3005 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [15:27:47] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [15:27:55] PROBLEM - graphoid endpoints health on scb1001 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:28:01] PROBLEM - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:28:17] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received https://wikitech.wikimedia.org/wiki/Citoid [15:28:35] PROBLEM - graphoid endpoints health on scb1004 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:28:43] PROBLEM - graphoid endpoints health on scb1002 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:28:53] PROBLEM - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:29:09] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:29:25] RECOVERY - PyBal connections to etcd on lvs3005 is OK: OK: 12 connections established with conf1006.eqiad.wmnet:4001 (min=12) https://wikitech.wikimedia.org/wiki/PyBal [15:29:41] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [15:29:41] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 4.448 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [15:30:09] PROBLEM - Restbase edge esams on text-lb.esams.wikimedia.org is CRITICAL: /api/rest_v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /api/rest_v1/transform/wikitext/to/html/{title} (Transform wikitext to html) timed out before a response was received: /api/rest_v1/media/math/check/{type} (Mathoid - check test formula) timed out before a response was [15:30:09] st_v1/page/mobile-html/{title} (Get mobile-html from storage) timed out before a response was received: /api/rest_v1/page/mobile-sections/{title} (Get mobile-sections for a test page on enwiki) timed out before a response was received: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before a response was received: /api/rest_v1/feed/announcements (Retrieve announcements) timed out before a response was re [15:30:09] _v1/page/references/{title} (Get references from storage) timed out before a response was received: /api/rest_v1/page/media-list/{title} (Get media-list from storage) timed out before a response was received: /api/rest_v1/page/html/{title} (Get html by title from storage) timed out before a response was received: /api/rest_v1/page/summary/{title} (Get summary from storage) timed out before a response was received: /api/rest_v1/pa [15:30:09] Get structured talk page for enwiki Salt article) timed out before a response was received: /api/rest_v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase [15:30:25] RECOVERY - graphoid endpoints health on scb1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:30:31] RECOVERY - graphoid endpoints health on scb1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:30:41] RECOVERY - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 563 bytes in 4.716 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:30:43] PROBLEM - SSH on lvs3005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:30:53] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15197 bytes in 0.592 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:31:11] RECOVERY - PyBal backends health check on lvs3005 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [15:31:37] RECOVERY - graphoid endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [15:32:33] RECOVERY - SSH on lvs3005 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [15:33:33] RECOVERY - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 4.035 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [15:33:37] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [15:33:51] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Citoid [15:37:03] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 77.4 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [15:43:10] !log 3*prepend in esams/knams [15:43:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:45:25] PROBLEM - Varnish traffic drop between 30min ago and now at codfw on icinga1001 is CRITICAL: 25.87 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [15:54:55] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is CRITICAL: 36.56 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [15:56:10] ^ expected [16:00:21] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is CRITICAL: 57.38 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:01:13] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received https://wikitech.wikimedia.org/wiki/Citoid [16:01:43] RECOVERY - Varnish traffic drop between 30min ago and now at codfw on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:01:45] PROBLEM - wiki content on commons #page on commons.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/project/view/1118/ [16:02:05] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:02:51] PROBLEM - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:03:07] PROBLEM - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:03:35] RECOVERY - wiki content on commons #page on commons.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 170470 bytes in 0.459 second response time https://phabricator.wikimedia.org/project/view/1118/ [16:03:41] PROBLEM - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:04:33] PROBLEM - Restbase edge esams on text-lb.esams.wikimedia.org is CRITICAL: WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by ProtocolError(Connection aborted., ConnectionResetError(104, Connection reset by peer)): /api/rest_v1/?spec https://wikitech.wikimedia.org/wiki/RESTBase [16:04:50] Hmm [16:04:54] I'm still seeing problems in esams [16:04:58] Erattic again for me [16:04:59] RECOVERY - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15183 bytes in 6.098 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:05:18] SSH is fine to labs hosts in eqiad but wikipedia and phab etc are struggling [16:06:05] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - testlb_443: Servers cp3050.esams.wmnet, cp3054.esams.wmnet, cp3064.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb_443: Servers cp3050.esams.wmnet, cp3062.esams.wmnet, cp3058.esams.wmnet, cp3052.esams.wmnet are marked down but pooled: testlb6_443: Servers cp3060.esams.wmnet, cp3050.esams.wmnet, cp3054.esams.wmnet, cp3062.e [16:06:05] 2.esams.wmnet, cp3064.esams.wmnet are marked down but pooled: textlb6_443: Servers cp3054.esams.wmnet, cp3062.esams.wmnet, cp3064.esams.wmnet, cp3058.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [16:06:47] * Krenair will be back later [16:06:58] krenair we're still working on the issue [16:07:01] k [16:07:17] RECOVERY - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 563 bytes in 0.167 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:07:27] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [16:07:41] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.847 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:08:03] RECOVERY - PyBal backends health check on lvs3005 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:08:33] RECOVERY - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 8.877 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:10:39] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Citoid [16:11:41] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on icinga1001 is OK: (C)60 le (W)70 le 74.31 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:13:43] PROBLEM - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:13:43] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - ncredirlb6_443: Servers ncredir3002.esams.wmnet are marked down but pooled: testlb_443: Servers cp3060.esams.wmnet, cp3050.esams.wmnet, cp3054.esams.wmnet, cp3062.esams.wmnet, cp3058.esams.wmnet, cp3052.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb_443: Servers cp3060.esams.wmnet, cp3064.esams.wmnet, cp3058.esams.wmnet, [16:13:43] t are marked down but pooled: testlb6_443: Servers cp3060.esams.wmnet, cp3058.esams.wmnet, cp3062.esams.wmnet, cp3052.esams.wmnet are marked down but pooled: textlb6_443: Servers cp3060.esams.wmnet, cp3054.esams.wmnet, cp3062.esams.wmnet, cp3058.esams.wmnet, cp3056.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [16:13:49] PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 39.07 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:14:55] PROBLEM - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:15:17] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:15:33] RECOVERY - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 4.350 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:16:30] Intresting... [16:16:43] RECOVERY - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 562 bytes in 1.528 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:16:46] I'm getting some connectivity issue with other Uk websites as well [16:17:31] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - testlb_443: Servers cp3060.esams.wmnet, cp3050.esams.wmnet, cp3064.esams.wmnet, cp3058.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: textlb_443: Servers cp3054.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: ncredirlb_443: Servers ncredir3001.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki [16:18:36] hi [16:18:56] zorun, hello, known problems, updates in #wikimedia-tech [16:19:03] I bet you know about the IPv6 issue with text-lb.esams.wikimedia.org; do you need any additional input? [16:19:15] RECOVERY - PyBal backends health check on lvs3005 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:19:25] no, but that you for your report. [16:19:29] *thank you [16:19:50] ok [16:21:15] PROBLEM - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:24:29] PROBLEM - Restbase edge esams on text-lb.esams.wikimedia.org is CRITICAL: /api/rest_v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received: /api/rest_v1/media/math/check/{type} (Mathoid - check test formula) timed out before a response was received: /api/rest_v1/page/media-list/{title} (Get media-list from storage) timed out before a response was received: /api/rest_v1 [16:24:29] yyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /api/rest_v1/feed/announcements (Retrieve announcements) timed out before a response was received: /api/rest_v1/transform/wikitext/to/html/{title} (Transform wikitext to html) timed out before a response was received: /api/rest_v1/page/mobile-html/{title} (Get mobile-html from storage) timed out before a response was rec [16:24:29] v1/page/html/{title} (Get html by title from storage) timed out before a response was received: /api/rest_v1/page/talk/{title} (Get structured talk page for enwiki Salt article) timed out before a response was received: /api/rest_v1/page/mobile-sections/{title} (Get mobile-sections for a test page on enwiki) timed out before a response was received: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before [16:24:29] ceived https://wikitech.wikimedia.org/wiki/RESTBase [16:25:07] PROBLEM - WDQS high update lag on wdqs1010 is CRITICAL: 3640 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [16:25:23] PROBLEM - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:25:49] PROBLEM - PyBal connections to etcd on lvs3005 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.20.0.15: Connection reset by peer https://wikitech.wikimedia.org/wiki/PyBal [16:25:53] PROBLEM - LVS HTTP IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:26:27] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15176 bytes in 1.459 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:26:47] RECOVERY - LVS HTTPS IPv4 #page on ncredir-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 3.399 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:29:07] RECOVERY - LVS HTTPS IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 233 bytes in 7.377 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:29:31] RECOVERY - LVS HTTP IPv6 #page on ncredir-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 159 bytes in 0.520 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:29:59] PROBLEM - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:30:33] PROBLEM - PyBal backends health check on lvs3005 is CRITICAL: PYBAL CRITICAL - CRITICAL - ncredirlb6_443: Servers ncredir3001.esams.wmnet are marked down but pooled: testlb_443: Servers cp3060.esams.wmnet are marked down but pooled: textlb_443: Servers cp3062.esams.wmnet, cp3056.esams.wmnet are marked down but pooled: ncredirlb_443: Servers ncredir3001.esams.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyB [16:31:33] RECOVERY - PyBal connections to etcd on lvs3005 is OK: OK: 12 connections established with conf1006.eqiad.wmnet:4001 (min=12) https://wikitech.wikimedia.org/wiki/PyBal [16:31:40] RECOVERY - LVS HTTP IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 563 bytes in 0.234 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:32:21] RECOVERY - PyBal backends health check on lvs3005 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:32:31] RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [16:33:45] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [16:37:15] PROBLEM - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 2562 bytes in 0.347 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:38:11] <_joe_> oh well [16:38:52] !log applying GRE MTU mitigation from T232602 to all cp1, cp3, cp5 cache nodes [16:38:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:38:56] T232602: GRE MTU mitigations - Tracking - https://phabricator.wikimedia.org/T232602 [16:42:40] !log ✔️ cdanis@cp4030.ulsfo.wmnet ~ 🕦☕ sudo depool [16:42:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:44:29] RECOVERY - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15183 bytes in 0.555 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [16:55:40] !log reduce /proc/sys/net/ipv4/tcp_synack_retries to 1 on esams text caches [16:55:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:03:10] !log reduce /proc/sys/net/ipv4/tcp_max_syn_backlog to 8192 on esams text caches [17:03:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:14:33] PROBLEM - Host elastic2043 is DOWN: PING CRITICAL - Packet loss = 100% [17:16:23] RECOVERY - Host elastic2043 is UP: PING WARNING - Packet loss = 37%, RTA = 36.17 ms [17:25:33] !log restart varnishkafka-webrequest on cp3056 [17:25:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:02] !log restart varnishkafka-webrequest on cp3064 [17:28:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:41:49] PROBLEM - Host elastic2043 is DOWN: PING CRITICAL - Packet loss = 100% [17:43:55] RECOVERY - Host elastic2043 is UP: PING OK - Packet loss = 0%, RTA = 36.15 ms [17:44:37] this host is rebooting, opening a task and notifying search team [17:57:59] 10Operations, 10ops-codfw, 10Discovery: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 (10Volans) p:05Triage→03Normal [18:01:17] !log volans@cumin1001 conftool action : set/pooled=inactive; selector: name=elastic2043.codfw.wmnet [18:01:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:01:39] !log depooled elastic2043 - T243715 [18:01:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:01:42] T243715: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 [18:07:07] 10Operations, 10ops-codfw, 10Discovery: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 (10Volans) Downtimed host until 2020-02-06 17:04:23 (no onsite dcops this week) [18:10:30] 10Operations, 10ops-codfw, 10Discovery: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 (10Volans) As per https://wikitech.wikimedia.org/wiki/Search#Hardware_Failures I'm depooling + downtiming + shutting down the host to avoid that it keeps rebooting and leav... [18:11:44] !log shutdown elastic2043 - T243715 [18:11:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:11:47] T243715: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 [18:16:17] 10Operations, 10ops-codfw, 10Discovery: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 (10Volans) Current status: ` /admin1-> racadm serveraction powerstatus Server power status: OFF ` [19:17:02] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Vachovec1) [19:21:36] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10CDanis) If you are still currently experiencing connectivity issues, please let us know, and when you do, please also report: - The IP address to which you resolve en.wikiped... [19:49:04] 10Operations, 10Discovery, 10Traffic, 10Wikidata, and 2 others: Wikidata queryservice lag repeatedly over 5s since Jan20, 2020 - https://phabricator.wikimedia.org/T243701 (10Dvorapa) [19:52:33] PROBLEM - MediaWiki memcached error rate on icinga1001 is CRITICAL: 1.233e+04 gt 5000 https://wikitech.wikimedia.org/wiki/Memcached https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=1&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [19:53:57] PROBLEM - High average GET latency for mw requests on appserver in eqiad on icinga1001 is CRITICAL: cluster=appserver code={200,204} handler=proxy:unix:/run/php/fpm-www.sock https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=appserver&var-method= [19:54:05] PROBLEM - PHP7 rendering on mw1319 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:07] PROBLEM - PHP7 rendering on mw1320 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:07] PROBLEM - Nginx local proxy to apache on mw1332 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:09] PROBLEM - Nginx local proxy to apache on mw1275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:09] PROBLEM - Nginx local proxy to apache on mw1269 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:09] PROBLEM - Apache HTTP on mw1264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:09] PROBLEM - Apache HTTP on mw1330 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:11] PROBLEM - Nginx local proxy to apache on mw1329 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:11] PROBLEM - Apache HTTP on mw1268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:13] PROBLEM - Apache HTTP on mw1270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:13] PROBLEM - Nginx local proxy to apache on mw1320 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:13] PROBLEM - Nginx local proxy to apache on mw1263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:13] PROBLEM - Apache HTTP on mw1262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:13] PROBLEM - PHP7 rendering on mw1326 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:15] PROBLEM - PHP7 rendering on mw1272 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:25] PROBLEM - Nginx local proxy to apache on mw1268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:25] PROBLEM - High average GET latency for mw requests on api_appserver in eqiad on icinga1001 is CRITICAL: cluster=api_appserver code=200 handler=proxy:unix:/run/php/fpm-www.sock https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=api_appserver&var-m [19:54:27] PROBLEM - Apache HTTP on mw1319 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:27] PROBLEM - Apache HTTP on mw1275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:31] PROBLEM - Apache HTTP on mw1324 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:31] PROBLEM - Nginx local proxy to apache on mw1248 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:35] PROBLEM - Apache HTTP on mw1250 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:35] PROBLEM - Apache HTTP on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:35] PROBLEM - Nginx local proxy to apache on mw1324 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:35] PROBLEM - Nginx local proxy to apache on mw1274 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:37] PROBLEM - PHP7 rendering on mw1333 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:39] PROBLEM - PHP7 rendering on mw1270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:39] PROBLEM - Apache HTTP on mw1261 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:45] PROBLEM - PHP7 rendering on mw1252 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:47] PROBLEM - Nginx local proxy to apache on mw1262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:47] PROBLEM - Nginx local proxy to apache on mw1271 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:47] PROBLEM - Nginx local proxy to apache on mw1325 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:51] PROBLEM - PHP7 rendering on mw1328 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:51] PROBLEM - Nginx local proxy to apache on mw1333 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:51] PROBLEM - Apache HTTP on mw1249 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:53] PROBLEM - PHP7 rendering on mw1248 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:53] PROBLEM - Apache HTTP on mw1347 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:53] PROBLEM - PHP7 rendering on mw1251 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:53] PROBLEM - Apache HTTP on mw1238 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:53] PROBLEM - PHP7 rendering on mw1239 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:57] PROBLEM - Nginx local proxy to apache on mw1326 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:54:57] PROBLEM - PHP7 rendering on mw1257 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:57] PROBLEM - PHP7 rendering on mw1325 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:57] PROBLEM - PHP7 rendering on mw1268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:57] PROBLEM - PHP7 rendering on mw1269 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:57] PROBLEM - PHP7 rendering on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:58] PROBLEM - PHP7 rendering on mw1263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:58] PROBLEM - PHP7 rendering on mw1274 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:59] PROBLEM - PHP7 rendering on mw1275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:54:59] PROBLEM - Apache HTTP on mw1329 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:00] PROBLEM - Nginx local proxy to apache on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:00] PROBLEM - Nginx local proxy to apache on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:01] PROBLEM - PHP7 rendering on mw1331 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:01] PROBLEM - Apache HTTP on mw1265 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:02] PROBLEM - Apache HTTP on mw1269 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:02] PROBLEM - PHP7 rendering on mw1273 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:03] PROBLEM - Apache HTTP on mw1325 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:03] PROBLEM - Apache HTTP on mw1271 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:04] PROBLEM - PHP7 rendering on mw1266 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:04] PROBLEM - Apache HTTP on mw1248 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:05] PROBLEM - PHP7 rendering on mw1230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:05] PROBLEM - PHP7 rendering on mw1254 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:06] PROBLEM - PHP7 rendering on mw1329 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:06] PROBLEM - PHP7 rendering on mw1330 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:07] PROBLEM - PHP7 rendering on mw1261 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:07] PROBLEM - Nginx local proxy to apache on mw1267 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:08] PROBLEM - Nginx local proxy to apache on mw1327 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:08] PROBLEM - Nginx local proxy to apache on mw1328 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:09] PROBLEM - Apache HTTP on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:09] PROBLEM - Apache HTTP on mw1288 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:10] PROBLEM - Apache HTTP on mw1281 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:10] PROBLEM - Nginx local proxy to apache on mw1330 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:11] PROBLEM - Nginx local proxy to apache on mw1264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:11] PROBLEM - LVS HTTPS IPv4 #page on text-lb.codfw.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [19:55:12] PROBLEM - PHP7 rendering on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:12] PROBLEM - PHP7 rendering on mw1278 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:13] PROBLEM - Apache HTTP on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:13] PROBLEM - PHP7 rendering on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:14] PROBLEM - Apache HTTP on mw1278 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:14] PROBLEM - Nginx local proxy to apache on mw1227 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:15] PROBLEM - Nginx local proxy to apache on mw1247 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:15] PROBLEM - Apache HTTP on mw1320 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:16] PROBLEM - PHP7 rendering on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:16] PROBLEM - PHP7 rendering on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:17] PROBLEM - PHP7 rendering on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:17] PROBLEM - Nginx local proxy to apache on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:18] PROBLEM - Nginx local proxy to apache on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:18] PROBLEM - proton endpoints health on proton2001 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Respond file not [19:55:19] xistent title) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [19:55:19] PROBLEM - Apache HTTP on mw1327 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:20] PROBLEM - Nginx local proxy to apache on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:20] PROBLEM - Apache HTTP on mw1272 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:23] PROBLEM - Nginx local proxy to apache on mw1270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:23] PROBLEM - Nginx local proxy to apache on mw1266 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:23] PROBLEM - Nginx local proxy to apache on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:23] PROBLEM - PHP7 rendering on mw1243 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:23] PROBLEM - PHP7 rendering on mw1262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:23] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - appservers-https_443: Servers mw1265.eqiad.wmnet, mw1238.eqiad.wmnet, mw1240.eqiad.wmnet, mw1246.eqiad.wmnet, mw1253.eqiad.wmnet, mw1267.eqiad.wmnet, mw1322.eqiad.wmnet, mw1323.eqiad.wmnet, mw1249.eqiad.wmnet, mw1327.eqiad.wmnet, mw1328.eqiad.wmnet, mw1243.eqiad.wmnet, mw1245.eqiad.wmnet, mw1272.eqiad.wmnet, mw1258.eqiad.wmnet, mw1329.eqiad. [19:55:24] ad.wmnet, mw1271.eqiad.wmnet, mw1264.eqiad.wmnet, mw1250.eqiad.wmnet, mw1266.eqiad.wmnet, mw1326.eqiad.wmnet, mw1268.eqiad.wmnet, mw1256.eqiad.wmnet, mw1333.eqiad.wmnet, mw1241.eqiad.wmnet, mw1255.eqiad.wmnet, mw1330.eqiad.wmnet, mw1257.eqiad.wmnet, mw1244.eqiad.wmnet, mw1331.eqiad.wmnet, mw1321.eqiad.wmnet, mw1269.eqiad.wmnet, mw1325.eqiad.wmnet, mw1254.eqiad.wmnet, mw1252.eqiad.wmnet, mw1261.eqiad.wmnet, mw1270.eqiad.wmnet, mw1 [19:55:24] mw1273.eqiad.wmnet, mw1262.eqiad.wmnet, mw1332.eqiad.wmnet, mw1247.eqiad.wmnet, mw1275.eqiad.wmnet are marked down but pooled: api_80: Servers mw1232.eqiad.wmnet, mw1344.eqiad.wmnet, mw https://wikitech.wikimedia.org/wiki/PyBal [19:55:25] PROBLEM - PHP7 rendering on mw1322 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:25] PROBLEM - proton endpoints health on proton1001 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Respond file not [19:55:26] xistent title) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [19:55:26] PROBLEM - proton endpoints health on proton2002 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received: /{domain}/v1/pdf/{title}/{format}/{type} (Respond file not [19:55:27] xistent title) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [19:55:27] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - appservers-https_443: Servers mw1265.eqiad.wmnet, mw1274.eqiad.wmnet, mw1242.eqiad.wmnet, mw1240.eqiad.wmnet, mw1246.eqiad.wmnet, mw1253.eqiad.wmnet, mw1267.eqiad.wmnet, mw1322.eqiad.wmnet, mw1238.eqiad.wmnet, mw1333.eqiad.wmnet, mw1323.eqiad.wmnet, mw1249.eqiad.wmnet, mw1327.eqiad.wmnet, mw1328.eqiad.wmnet, mw1243.eqiad.wmnet, mw1245.eqiad. [19:55:28] ad.wmnet, mw1263.eqiad.wmnet, mw1258.eqiad.wmnet, mw1329.eqiad.wmnet, mw1320.eqiad.wmnet, mw1271.eqiad.wmnet, mw1264.eqiad.wmnet, mw1250.eqiad.wmnet, mw1266.eqiad.wmnet, mw1326.eqiad.wmnet, mw1268.eqiad.wmnet, mw1256.eqiad.wmnet, mw1319.eqiad.wmnet, mw1241.eqiad.wmnet, mw1324.eqiad.wmnet, mw1255.eqiad.wmnet, mw1257.eqiad.wmnet, mw1251.eqiad.wmnet, mw1239.eqiad.wmnet, mw1244.eqiad.wmnet, mw1331.eqiad.wmnet, mw1321.eqiad.wmnet, mw1 [19:55:28] mw1325.eqiad.wmnet, mw1254.eqiad.wmnet, mw1248.eqiad.wmnet, mw1252.eqiad.wmnet, mw1261.eqiad.wmnet, mw1270.eqiad.wmnet, mw1330.eqiad.wmnet, mw1273.eqiad.wmnet, mw1262.eqiad.wmnet, mw133 https://wikitech.wikimedia.org/wiki/PyBal [19:55:29] PROBLEM - PHP7 rendering on mw1288 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:29] PROBLEM - PHP7 rendering on mw1340 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:30] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [19:55:30] PROBLEM - High average POST latency for mw requests on api_appserver in eqiad on icinga1001 is CRITICAL: cluster=api_appserver code=200 handler=proxy:unix:/run/php/fpm-www.sock https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=api_appserver&var- [19:55:31] PROBLEM - High average POST latency for mw requests on appserver in eqiad on icinga1001 is CRITICAL: cluster=appserver code=200 handler=proxy:unix:/run/php/fpm-www.sock https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=appserver&var-method=POST [19:55:31] PROBLEM - PHP7 rendering on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:32] PROBLEM - PHP7 rendering on mw1229 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:32] PROBLEM - PHP7 rendering on mw1231 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:33] PROBLEM - PHP7 rendering on mw1264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:33] PROBLEM - PHP7 rendering on mw1339 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:34] PROBLEM - LVS HTTPS IPv6 #page on text-lb.eqsin.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [19:55:34] PROBLEM - Nginx local proxy to apache on mw1322 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:35] PROBLEM - Apache HTTP on mw1223 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:35] PROBLEM - Apache HTTP on mw1231 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:36] PROBLEM - PHP7 rendering on mw1284 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:36] PROBLEM - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [19:55:37] PROBLEM - Apache HTTP on mw1254 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:37] PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: /{domain}/v1/page/media/{title} (Get media in test page) timed out before a response was received: /{domain}/v1/page/metadata/{title} (retrieve extended metadata for Video article on English Wikipedia) timed out before a response was received: /{domain}/v1/page/mobile-sections/{title} (retrieve test page via mobile-sections) timed out before a response was received: /{ [19:55:38] obile-html/{title} (Get page content HTML for test page) timed out before a response was received: /{domain}/v1/page/summary/{title} (Get summary for test page) timed out before a response was received: /{domain}/v1/transform/html/to/mobile-html/{title} (Get preview mobile HTML for test page) timed out before a response was received: /{domain}/v1/page/random/title (retrieve a random article title) timed out before a response was [19:55:38] n}/v1/page/media-list/{title} (Get media list from test page) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [19:55:39] PROBLEM - Apache HTTP on mw1345 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:39] PROBLEM - Apache HTTP on mw1240 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:40] PROBLEM - PHP7 rendering on mw1265 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [19:55:40] PROBLEM - Nginx local proxy to apache on mw1223 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:41] PROBLEM - termbox codfw on termbox.svc.codfw.wmnet is CRITICAL: /termbox (get rendered termbox) is CRITICAL: Test get rendered termbox returned the unexpected status 500 (expecting: 200) https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/SSR_Service [19:55:41] PROBLEM - Nginx local proxy to apache on mw1226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [19:55:46] rip [19:56:05] what the?! [19:56:17] @bradv basically a lot just sorta crashed [19:56:29] lot of stuff went "CRITICAL" [19:56:34] just before you joined [19:56:59] Something just got broken [19:57:02] Okay, it's not just me then. [19:57:15] Fortunately, fixing it is not my problem. [19:57:16] looks like something lost power(?) [19:57:27] 10:16 AM <+wikibugs> Operations, ops-codfw, Discovery: elastic2043 has hardware errors that trigger reboots - https://phabricator.wikimedia.org/T243715 (Volans) Current status: ` /admin1-> racadm serveraction powerstatus Server power status: OFF ` [19:57:43] (I know I started it, but let's keep this channel clear for the people who /do/ need to fix it) [19:57:54] random comments do not help [19:58:10] people are looking into it right now [19:58:17] just to confirm: SRE team is actively working on one or more incidents, we're aware of everything [20:00:27] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Piastu) Łódź, Poland Some 504 Gateway Time-out for a while After that tracert looks like this: ` traceroute to en.wikipedia.org (91.198.174.192), 30 hops max, 60 byte packets 1... [20:00:42] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Piastu) Łódź, Poland Some 504 Gateway Time-out for a while After that tracert looks like this: ` traceroute to en.wikipedia.org (91.198.174.192), 30 hops max, 60 byte packets 1... [20:03:05] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Bawolff) Im experiancing significant slowness (but it eventually works) connecting to 198.35.26.96 (dyna.wikimedia.org) [on a cellphone, cant easily get a traceroute] [20:06:56] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Ankit-Maity) p:05Triage→03Unbreak! Got an error page (Error 504) once and now site does not load at all. My traceroute is completely empty: ` traceroute to dyna.wikimedia.o... [20:13:27] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Piastu) And i've got one more tracert from someone else — Warsaw, Poland: ` Tracing route to www.wikipedia.pl [94.23.242.48] over a maximum of 30 hops: 1 3 ms 2 ms 2 ms 192.168.... [20:13:35] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Piastu) And i've got one more tracert from someone else — Warsaw, Poland: ` Tracing route to www.wikipedia.pl [94.23.242.48] over a maximum of 30 hops: 1 3 ms 2 ms 2 ms 192.168.... [20:18:34] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10ShakespeareFan00) Adding a trace (London, UK) ` 01/26/20 20:13:36 Slow traceroute 91.198.174.192 Trace 91.198.174.192 ... [[REDACTED]] RTT: 1ms TTL: 64 [[REDACTED]] 62.3.... [20:19:04] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Riley_Huntley) ` Tracing route to dyna.wikimedia.org [198.35.26.96] over a maximum of 30 hops: 1 <1 ms <1 ms 1 ms 192.168.0.1 2 13 ms 10 ms 15 ms 147.1... [20:27:39] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Pigsonthewing) From Birmingham, England just now: Tracing route to dyna.wikimedia.org [91.198.174.192] over a maximum of 30 hops: 1 3 ms 3 ms 5 ms BrightBox.ee... [20:28:58] commons appears to be quite dead [20:29:20] sladen, Known issue: operations is aware and working on it. Join #wikimedia-tech for updates. [20:29:42] AntiComposite: danke! [20:32:18] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10TheSandDoctor) Stopped mine manually. Looks like it is down here too. British Columbia if it matters. ` traceroute en.wikipedia.org traceroute to dyna.wikimedia.org (198.35.26.9... [20:35:15] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10RhinosF1) Norway: https://www.irccloud.com/pastebin/kGcHnxXE US EAST COAST: https://imgur.com/oljPECR [20:39:16] RECOVERY - Apache HTTP on mw1247 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 5.727 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:18] PROBLEM - Nginx local proxy to apache on mw1234 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:39:20] RECOVERY - Apache HTTP on mw1297 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.303 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:22] RECOVERY - LVS HTTPS IPv6 #page on text-lb.codfw.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15197 bytes in 4.778 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:39:22] RECOVERY - PHP7 rendering on mw1250 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 5.737 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:39:24] PROBLEM - Nginx local proxy to apache on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:39:24] RECOVERY - Apache HTTP on mw1317 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.253 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:28] RECOVERY - Nginx local proxy to apache on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.619 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:28] PROBLEM - Apache HTTP on mw1281 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:39:28] PROBLEM - Apache HTTP on mw1226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:39:34] RECOVERY - PHP7 rendering on mw1228 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 9.804 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:39:36] PROBLEM - PHP7 rendering on mw1343 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:39:36] RECOVERY - PHP7 rendering on mw1227 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 6.034 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:39:36] RECOVERY - Nginx local proxy to apache on mw1254 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.048 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:38] RECOVERY - Nginx local proxy to apache on mw1229 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.769 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:40] PROBLEM - Nginx local proxy to apache on mw1284 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:39:40] RECOVERY - Nginx local proxy to apache on mw1287 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.479 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:39:50] PROBLEM - Apache HTTP on mw1324 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:39:56] PROBLEM - Nginx local proxy to apache on mw1248 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:02] PROBLEM - PHP7 rendering on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:04] PROBLEM - Apache HTTP on mw1230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:06] RECOVERY - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15185 bytes in 8.810 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:40:06] PROBLEM - Nginx local proxy to apache on mw1251 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:08] PROBLEM - Apache HTTP on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:10] PROBLEM - Apache HTTP on mw1246 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:10] PROBLEM - PHP7 rendering on mw1246 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:10] PROBLEM - Nginx local proxy to apache on mw1325 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:10] PROBLEM - Apache HTTP on mw1331 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:12] PROBLEM - Nginx local proxy to apache on mw1331 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:12] PROBLEM - Nginx local proxy to apache on mw1282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:14] PROBLEM - PHP7 rendering on mw1328 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:15] PROBLEM - PHP7 rendering on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:20] RECOVERY - Apache HTTP on mw1228 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.365 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:40:20] PROBLEM - Nginx local proxy to apache on mw1339 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:20] PROBLEM - Apache HTTP on mw1326 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:22] PROBLEM - Nginx local proxy to apache on mw1261 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:22] PROBLEM - Apache HTTP on mw1263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:22] PROBLEM - PHP7 rendering on mw1263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:22] PROBLEM - Apache HTTP on mw1274 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:22] PROBLEM - Apache HTTP on mw1265 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:22] PROBLEM - Apache HTTP on mw1340 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:24] PROBLEM - PHP7 rendering on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:24] PROBLEM - PHP7 rendering on mw1274 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:28] RECOVERY - LVS HTTPS IPv4 #page on text-lb.codfw.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15186 bytes in 7.464 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:40:28] PROBLEM - PHP7 rendering on mw1266 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:28] PROBLEM - Apache HTTP on mw1271 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:28] RECOVERY - PHP7 rendering on mw1313 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 7.748 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:28] PROBLEM - Apache HTTP on mw1264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:28] PROBLEM - Nginx local proxy to apache on mw1264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:29] PROBLEM - Nginx local proxy to apache on mw1269 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:30] PROBLEM - Nginx local proxy to apache on mw1328 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:30] PROBLEM - Nginx local proxy to apache on mw1330 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:30] PROBLEM - PHP7 rendering on mw1330 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:30] RECOVERY - Nginx local proxy to apache on mw1290 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.187 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:40:31] RECOVERY - PHP7 rendering on mw1242 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 6.157 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:31] PROBLEM - Nginx local proxy to apache on mw1332 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:32] PROBLEM - Apache HTTP on mw1330 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:34] RECOVERY - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15186 bytes in 7.751 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:40:34] RECOVERY - PHP7 rendering on mw1341 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 9.590 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:34] PROBLEM - Apache HTTP on mw1268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:35] PROBLEM - Apache HTTP on mw1270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:35] PROBLEM - Nginx local proxy to apache on mw1263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:36] PROBLEM - PHP7 rendering on mw1272 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:36] PROBLEM - Apache HTTP on mw1272 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:36] PROBLEM - Nginx local proxy to apache on mw1270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:36] PROBLEM - PHP7 rendering on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:44] PROBLEM - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is CRITICAL: /api/rest_v1/page/title/{title} (Get rev by title from storage) timed out before a response was received: /api/rest_v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase [20:40:46] PROBLEM - Nginx local proxy to apache on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:52] PROBLEM - PHP7 rendering on mw1282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:40:52] PROBLEM - Apache HTTP on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:40:57] PROBLEM - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:41:00] PROBLEM - Apache HTTP on mw1241 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:41:02] PROBLEM - Nginx local proxy to apache on mw1314 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:41:02] PROBLEM - PHP7 rendering on mw1231 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:41:06] RECOVERY - Apache HTTP on mw1224 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.195 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:08] PROBLEM - PHP7 rendering on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:41:12] PROBLEM - Nginx local proxy to apache on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:41:18] PROBLEM - Apache HTTP on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:41:18] RECOVERY - Apache HTTP on mw1283 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.826 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:20] RECOVERY - Nginx local proxy to apache on mw1278 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.187 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:20] PROBLEM - Apache HTTP on mw1316 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:41:24] PROBLEM - Nginx local proxy to apache on mw1326 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:41:26] RECOVERY - Nginx local proxy to apache on mw1222 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.963 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:30] RECOVERY - Apache HTTP on mw1226 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.091 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:32] PROBLEM - PHP7 rendering on mw1278 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:41:36] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error - https://phabricator.wikimedia.org/T243713 (10Davey2010) From Rochester, UK right now. Tracing route to dyna.wikimedia.org [91.198.174.192] over a maximum of 30 hops: 1 3 ms 3 ms 3 ms [REDACTED] [[REDACTED]... [20:41:50] RECOVERY - Apache HTTP on mw1324 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.086 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:55] RECOVERY - Nginx local proxy to apache on mw1245 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.763 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:55] RECOVERY - Apache HTTP on mw1245 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.778 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:56] RECOVERY - Apache HTTP on mw1319 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.612 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:58] PROBLEM - LVS HTTPS IPv4 #page on text-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:41:58] RECOVERY - Apache HTTP on mw1321 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.613 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:58] RECOVERY - Apache HTTP on mw1345 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.852 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:41:58] PROBLEM - Nginx local proxy to apache on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:02] RECOVERY - Apache HTTP on mw1328 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.853 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:04] RECOVERY - PHP7 rendering on mw1233 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 9.627 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:08] RECOVERY - Nginx local proxy to apache on mw1251 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.697 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:10] RECOVERY - PHP opcache health on mw1333 is OK: OK: opcache is healthy https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_opcache_health [20:42:10] PROBLEM - Nginx local proxy to apache on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:12] PROBLEM - PHP7 rendering on mw1255 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:12] PROBLEM - Apache HTTP on mw1255 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:14] PROBLEM - PHP7 rendering on mw1252 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:18] PROBLEM - Apache HTTP on mw1333 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:20] RECOVERY - Nginx local proxy to apache on mw1286 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.712 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:22] PROBLEM - Nginx local proxy to apache on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:24] RECOVERY - Apache HTTP on mw1348 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.894 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:24] PROBLEM - Apache HTTP on mw1286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:24] PROBLEM - Nginx local proxy to apache on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:30] RECOVERY - Apache HTTP on mw1222 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.901 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:30] PROBLEM - Nginx local proxy to apache on mw1228 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:34] PROBLEM - Apache HTTP on mw1327 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:36] RECOVERY - PHP7 rendering on mw1222 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 9.398 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:36] PROBLEM - Nginx local proxy to apache on mw1320 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:36] PROBLEM - Apache HTTP on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:36] PROBLEM - PHP7 rendering on mw1326 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:38] PROBLEM - Apache HTTP on mw1262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:40] PROBLEM - PHP7 rendering on mw1262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:40] PROBLEM - Nginx local proxy to apache on mw1266 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:42] PROBLEM - PHP7 rendering on mw1264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:42] RECOVERY - Nginx local proxy to apache on mw1225 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.400 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:44] PROBLEM - Apache HTTP on mw1322 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:48] PROBLEM - PHP7 rendering on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:48] PROBLEM - Apache HTTP on mw1267 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:48] RECOVERY - Apache HTTP on mw1323 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.863 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:48] PROBLEM - Nginx local proxy to apache on mw1268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:48] RECOVERY - Nginx local proxy to apache on mw1343 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.451 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:48] PROBLEM - PHP7 rendering on mw1265 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:48] PROBLEM - Apache HTTP on mw1275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:50] PROBLEM - PHP7 rendering on mw1267 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:50] PROBLEM - PHP7 rendering on mw1327 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:50] PROBLEM - PHP7 rendering on mw1276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:52] RECOVERY - PHP7 rendering on mw1234 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.303 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:54] RECOVERY - Apache HTTP on mw1234 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.264 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:54] PROBLEM - PHP7 rendering on mw1333 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:42:54] PROBLEM - Apache HTTP on mw1266 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:54] PROBLEM - Nginx local proxy to apache on mw1274 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:42:56] RECOVERY - Apache HTTP on mw1241 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.396 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:42:58] RECOVERY - Nginx local proxy to apache on mw1314 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.489 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:00] PROBLEM - Apache HTTP on mw1250 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:00] PROBLEM - Nginx local proxy to apache on mw1239 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:02] RECOVERY - PHP7 rendering on mw1249 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 6.660 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:02] PROBLEM - Nginx local proxy to apache on mw1250 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:04] PROBLEM - PHP7 rendering on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:08] RECOVERY - Nginx local proxy to apache on mw1285 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.889 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:08] RECOVERY - Nginx local proxy to apache on mw1344 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.231 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:08] RECOVERY - Apache HTTP on mw1313 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.601 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:24] PROBLEM - Apache HTTP on mw1276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:24] PROBLEM - PHP7 rendering on mw1286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:24] PROBLEM - PHP7 rendering on mw1320 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:28] PROBLEM - Apache HTTP on mw1320 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:30] PROBLEM - PHP7 rendering on mw1221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:30] PROBLEM - Nginx local proxy to apache on mw1221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:32] RECOVERY - PHP7 rendering on mw1343 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.956 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:34] PROBLEM - PHP7 rendering on mw1322 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:43:44] PROBLEM - Apache HTTP on mw1256 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:50] RECOVERY - Apache HTTP on mw1223 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.778 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:52] RECOVERY - LVS HTTPS IPv4 #page on text-lb.eqiad.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15186 bytes in 7.854 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:43:52] RECOVERY - Nginx local proxy to apache on mw1348 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.161 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:52] RECOVERY - Nginx local proxy to apache on mw1322 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.016 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:52] RECOVERY - Nginx local proxy to apache on mw1248 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.831 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:43:54] PROBLEM - Nginx local proxy to apache on mw1241 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:54] PROBLEM - Apache HTTP on mw1346 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:43:58] RECOVERY - Apache HTTP on mw1230 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.845 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:06] RECOVERY - PHP7 rendering on mw1252 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 7.310 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:08] RECOVERY - Nginx local proxy to apache on mw1319 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.939 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:12] RECOVERY - Nginx local proxy to apache on mw1323 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.248 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:12] RECOVERY - Apache HTTP on mw1333 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.270 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:12] RECOVERY - Apache HTTP on mw1339 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.582 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:14] PROBLEM - PHP7 rendering on mw1324 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:15] RECOVERY - Nginx local proxy to apache on mw1339 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.527 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:16] PROBLEM - Apache HTTP on mw1297 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:16] RECOVERY - Apache HTTP on mw1326 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.473 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:16] RECOVERY - Nginx local proxy to apache on mw1289 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.019 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:16] RECOVERY - Apache HTTP on mw1286 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.193 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:18] PROBLEM - Apache HTTP on mw1290 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:24] RECOVERY - Nginx local proxy to apache on mw1228 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.371 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:24] PROBLEM - Nginx local proxy to apache on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:30] PROBLEM - Apache HTTP on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:30] PROBLEM - Apache HTTP on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:34] PROBLEM - PHP7 rendering on mw1227 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:34] PROBLEM - Apache HTTP on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:36] PROBLEM - Nginx local proxy to apache on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:40] RECOVERY - PHP7 rendering on mw1342 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.628 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:42] RECOVERY - PHP7 rendering on mw1276 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 7.901 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:44] RECOVERY - PHP7 rendering on mw1282 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.523 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:44] PROBLEM - Apache HTTP on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:44] RECOVERY - PHP7 rendering on mw1289 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.045 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:50] PROBLEM - PHP7 rendering on mw1253 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:52] PROBLEM - PHP7 rendering on mw1339 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:54] RECOVERY - Apache HTTP on mw1250 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.101 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:54] RECOVERY - PHP7 rendering on mw1231 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.683 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:44:56] RECOVERY - Nginx local proxy to apache on mw1239 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.701 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:56] RECOVERY - Nginx local proxy to apache on mw1250 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.161 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:44:56] PROBLEM - Nginx local proxy to apache on mw1313 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:56] PROBLEM - Nginx local proxy to apache on mw1317 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:44:58] PROBLEM - Nginx local proxy to apache on mw1324 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:45:02] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:45:08] RECOVERY - PHP7 rendering on mw1223 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 9.559 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:18] RECOVERY - PHP7 rendering on mw1286 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.856 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:18] PROBLEM - Nginx local proxy to apache on mw1257 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:45:18] PROBLEM - Apache HTTP on mw1228 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:45:20] RECOVERY - Apache HTTP on mw1276 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.903 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:45:20] PROBLEM - PHP7 rendering on mw1257 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:20] PROBLEM - PHP7 rendering on mw1245 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:22] RECOVERY - Apache HTTP on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.361 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:45:28] PROBLEM - PHP7 rendering on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:28] PROBLEM - Nginx local proxy to apache on mw1290 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:45:28] PROBLEM - Nginx local proxy to apache on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:45:32] PROBLEM - LVS HTTPS IPv6 #page on text-lb.ulsfo.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:45:32] PROBLEM - PHP7 rendering on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:32] PROBLEM - PHP7 rendering on mw1279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:50] RECOVERY - Apache HTTP on mw1346 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.774 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:45:52] PROBLEM - PHP7 rendering on mw1281 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:45:54] PROBLEM - Apache HTTP on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:45:58] PROBLEM - Nginx local proxy to apache on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:00] PROBLEM - Apache HTTP on mw1287 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:00] RECOVERY - Apache HTTP on mw1232 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.726 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:02] PROBLEM - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:46:02] PROBLEM - Apache HTTP on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:02] RECOVERY - Apache HTTP on mw1246 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.915 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:02] RECOVERY - PHP7 rendering on mw1246 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 7.114 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:46:04] RECOVERY - Nginx local proxy to apache on mw1276 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.605 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:04] PROBLEM - Nginx local proxy to apache on mw1240 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:08] RECOVERY - PHP7 rendering on mw1324 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.405 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:46:14] RECOVERY - Apache HTTP on mw1297 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.429 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:14] RECOVERY - Nginx local proxy to apache on mw1277 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.405 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:18] PROBLEM - Apache HTTP on mw1247 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:20] PROBLEM - Nginx local proxy to apache on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:22] PROBLEM - PHP7 rendering on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:46:22] RECOVERY - PHP7 rendering on mw1230 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 8.934 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:46:25] PROBLEM - Nginx local proxy to apache on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:26] RECOVERY - Apache HTTP on mw1312 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.232 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:26] RECOVERY - Apache HTTP on mw1289 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.579 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:28] RECOVERY - Apache HTTP on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.199 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:30] PROBLEM - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:46:40] RECOVERY - Apache HTTP on mw1275 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.534 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:42] RECOVERY - Apache HTTP on mw1285 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.923 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:46:46] PROBLEM - Apache HTTP on mw1314 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:48] RECOVERY - PHP7 rendering on mw1333 is OK: HTTP OK: HTTP/1.1 200 OK - 75937 bytes in 9.331 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:46:50] PROBLEM - Apache HTTP on mw1319 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:52] PROBLEM - Apache HTTP on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:46:56] PROBLEM - wikidata.org dispatch lag is REALLY high ---4000s- on www.wikidata.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/project/view/71/ [20:46:58] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15200 bytes in 7.243 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:46:58] PROBLEM - PHP7 rendering on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:02] RECOVERY - Nginx local proxy to apache on mw1246 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.633 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:02] RECOVERY - Apache HTTP on mw1258 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.723 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:04] RECOVERY - PHP7 rendering on mw1314 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.936 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:10] RECOVERY - Apache HTTP on mw1277 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.184 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:12] RECOVERY - Apache HTTP on mw1316 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.668 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:14] PROBLEM - PHP7 rendering on mw1239 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:14] PROBLEM - Nginx local proxy to apache on mw1286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:14] PROBLEM - PHP7 rendering on mw1346 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:14] RECOVERY - Nginx local proxy to apache on mw1234 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.330 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:14] RECOVERY - Nginx local proxy to apache on mw1257 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 4.600 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:15] RECOVERY - PHP7 rendering on mw1257 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 4.898 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:16] RECOVERY - Nginx local proxy to apache on mw1312 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.779 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:16] PROBLEM - PHP7 rendering on mw1248 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:16] PROBLEM - PHP7 rendering on mw1232 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:18] RECOVERY - PHP7 rendering on mw1245 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.502 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:18] PROBLEM - PHP7 rendering on mw1258 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:18] PROBLEM - Apache HTTP on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:20] PROBLEM - PHP7 rendering on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:23] PROBLEM - LVS HTTPS IPv6 #page on text-lb.codfw.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:47:23] PROBLEM - Apache HTTP on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:24] RECOVERY - PHP7 rendering on mw1329 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.780 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:26] PROBLEM - Apache HTTP on mw1288 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:28] PROBLEM - Apache HTTP on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:30] PROBLEM - PHP7 rendering on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:30] PROBLEM - Nginx local proxy to apache on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:34] RECOVERY - Nginx local proxy to apache on mw1284 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.962 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:35] PROBLEM - Nginx local proxy to apache on mw1329 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:38] RECOVERY - Nginx local proxy to apache on mw1227 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.342 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:40] RECOVERY - Apache HTTP on mw1256 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.574 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:40] PROBLEM - Apache HTTP on mw1279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:44] PROBLEM - Nginx local proxy to apache on mw1316 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:50] PROBLEM - PHP7 rendering on mw1229 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:52] RECOVERY - Nginx local proxy to apache on mw1241 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.798 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:54] RECOVERY - Apache HTTP on mw1341 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.674 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:47:58] PROBLEM - Nginx local proxy to apache on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:58] PROBLEM - PHP7 rendering on mw1290 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:58] PROBLEM - Nginx local proxy to apache on mw1281 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:47:58] PROBLEM - PHP7 rendering on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:47:58] RECOVERY - Apache HTTP on mw1287 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.776 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:00] RECOVERY - Nginx local proxy to apache on mw1232 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.151 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:00] RECOVERY - Nginx local proxy to apache on mw1235 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.687 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:00] RECOVERY - Apache HTTP on mw1255 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 5.614 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:00] RECOVERY - PHP7 rendering on mw1255 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 5.689 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:02] RECOVERY - LVS HTTP IPv4 #page on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 23765 bytes in 8.429 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:48:02] PROBLEM - Nginx local proxy to apache on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:02] PROBLEM - Nginx local proxy to apache on mw1344 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:04] PROBLEM - Apache HTTP on mw1313 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:04] RECOVERY - Nginx local proxy to apache on mw1240 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.754 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:08] RECOVERY - Nginx local proxy to apache on mw1325 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.037 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:10] PROBLEM - Apache HTTP on mw1249 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:10] PROBLEM - Nginx local proxy to apache on mw1333 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:14] RECOVERY - Nginx local proxy to apache on mw1340 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.637 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:14] RECOVERY - PHP7 rendering on mw1225 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.478 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:18] RECOVERY - Apache HTTP on mw1290 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.314 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:20] PROBLEM - Nginx local proxy to apache on mw1230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:22] PROBLEM - Apache HTTP on mw1317 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:24] RECOVERY - PHP7 rendering on mw1323 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.854 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:26] RECOVERY - Nginx local proxy to apache on mw1342 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.704 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:26] PROBLEM - Apache HTTP on mw1251 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:30] RECOVERY - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15185 bytes in 7.574 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:48:30] PROBLEM - PHP7 rendering on mw1343 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:32] PROBLEM - PHP7 rendering on mw1244 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:34] PROBLEM - Nginx local proxy to apache on mw1287 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:46] RECOVERY - Apache HTTP on mw1314 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.971 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:48] PROBLEM - Nginx local proxy to apache on mw1223 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:52] RECOVERY - LVS HTTPS IPv6 #page on text-lb.eqsin.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 8.619 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:48:54] RECOVERY - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15199 bytes in 9.309 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:48:55] PROBLEM - Nginx local proxy to apache on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:55] RECOVERY - Apache HTTP on mw1319 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.324 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:55] RECOVERY - PHP7 rendering on mw1253 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.332 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:55] PROBLEM - Apache HTTP on mw1223 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:55] PROBLEM - PHP7 rendering on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:55] PROBLEM - Nginx local proxy to apache on mw1322 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:48:55] RECOVERY - PHP7 rendering on mw1284 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.889 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:48:58] RECOVERY - Nginx local proxy to apache on mw1313 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.732 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:48:58] PROBLEM - Apache HTTP on mw1328 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:00] PROBLEM - Apache HTTP on mw1230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:10] RECOVERY - Apache HTTP on mw1227 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.707 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:49:10] PROBLEM - Nginx local proxy to apache on mw1319 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:18] RECOVERY - PHP7 rendering on mw1239 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.240 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:18] RECOVERY - PHP7 rendering on mw1346 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.162 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:18] PROBLEM - Apache HTTP on mw1326 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:20] RECOVERY - PHP7 rendering on mw1248 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.801 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:20] PROBLEM - Apache HTTP on mw1286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:22] RECOVERY - PHP7 rendering on mw1232 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.536 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:22] RECOVERY - PHP7 rendering on mw1258 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.665 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:28] RECOVERY - Apache HTTP on mw1222 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.676 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:49:28] PROBLEM - Apache HTTP on mw1244 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:30] RECOVERY - PHP7 rendering on mw1278 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.969 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:30] RECOVERY - Apache HTTP on mw1288 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.521 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:49:32] PROBLEM - PHP7 rendering on mw1283 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:34] RECOVERY - PHP7 rendering on mw1224 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.477 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:34] RECOVERY - Nginx local proxy to apache on mw1341 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.881 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:49:34] RECOVERY - PHP7 rendering on mw1222 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.064 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:40] RECOVERY - Apache HTTP on mw1279 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.885 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:49:42] PROBLEM - PHP7 rendering on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:46] RECOVERY - Nginx local proxy to apache on mw1316 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.547 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:49:46] PROBLEM - PHP7 rendering on mw1276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:46] PROBLEM - PHP7 rendering on mw1282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:50] PROBLEM - PHP7 rendering on mw1234 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:49:50] PROBLEM - Apache HTTP on mw1234 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:49:52] RECOVERY - PHP7 rendering on mw1229 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.025 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:50:00] RECOVERY - PHP7 rendering on mw1290 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.456 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:50:04] RECOVERY - Nginx local proxy to apache on mw1280 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.843 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:05] RECOVERY - Nginx local proxy to apache on mw1344 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.994 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:10] RECOVERY - Apache HTTP on mw1331 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.674 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:10] RECOVERY - Nginx local proxy to apache on mw1333 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.138 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:10] RECOVERY - Nginx local proxy to apache on mw1331 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.704 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:12] PROBLEM - PHP7 rendering on mw1223 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:50:12] RECOVERY - Nginx local proxy to apache on mw1279 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.634 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:12] PROBLEM - Apache HTTP on mw1343 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:50:16] RECOVERY - Apache HTTP on mw1247 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 5.680 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:16] PROBLEM - Apache HTTP on mw1248 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:50:16] PROBLEM - PHP7 rendering on mw1345 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:50:18] PROBLEM - PHP7 rendering on mw1286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:50:18] PROBLEM - Apache HTTP on mw1276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:50:20] RECOVERY - Nginx local proxy to apache on mw1230 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.003 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:20] RECOVERY - PHP7 rendering on mw1321 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.490 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:50:22] RECOVERY - Apache HTTP on mw1221 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.118 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:24] PROBLEM - Apache HTTP on mw1281 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:50:24] PROBLEM - Apache HTTP on mw1284 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:50:25] RECOVERY - Apache HTTP on mw1251 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.919 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:26] RECOVERY - Nginx local proxy to apache on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.657 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:32] RECOVERY - Apache HTTP on mw1235 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.827 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:56] RECOVERY - Apache HTTP on mw1230 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.448 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:50:58] RECOVERY - Nginx local proxy to apache on mw1324 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.898 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:00] RECOVERY - PHP7 rendering on mw1280 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.547 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:00] RECOVERY - PHP7 rendering on mw1235 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.823 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:00] RECOVERY - PHP7 rendering on mw1347 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.890 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:06] RECOVERY - Apache HTTP on mw1332 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.853 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:06] PROBLEM - Apache HTTP on mw1339 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:12] RECOVERY - Apache HTTP on mw1286 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.420 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:14] PROBLEM - WDQS high update lag on wdqs1009 is CRITICAL: 3613 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:51:18] PROBLEM - PHP7 rendering on mw1230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:18] PROBLEM - Nginx local proxy to apache on mw1228 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:20] RECOVERY - Nginx local proxy to apache on mw1346 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.259 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:22] PROBLEM - Nginx local proxy to apache on mw1238 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:22] PROBLEM - Apache HTTP on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:22] PROBLEM - Apache HTTP on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:24] PROBLEM - WDQS high update lag on wdqs1008 is CRITICAL: 3623 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:51:24] PROBLEM - PHP7 rendering on mw1228 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:24] RECOVERY - PHP7 rendering on mw1283 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.603 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:25] RECOVERY - Apache HTTP on mw1280 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.989 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:25] !log esams text caches: reverting earlier sysctl mitigations [20:51:26] PROBLEM - Nginx local proxy to apache on mw1229 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:26] PROBLEM - PHP7 rendering on mw1297 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:28] PROBLEM - Nginx local proxy to apache on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:34] RECOVERY - PHP7 rendering on mw1342 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 8.377 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:36] PROBLEM - Apache HTTP on mw1275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:38] PROBLEM - WDQS high update lag on wdqs1003 is CRITICAL: 3632 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:51:38] PROBLEM - Apache HTTP on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:40] PROBLEM - Apache HTTP on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:51:46] RECOVERY - PHP7 rendering on mw1281 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.813 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:51:50] RECOVERY - Nginx local proxy to apache on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.040 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:51:58] RECOVERY - Apache HTTP on mw1313 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.979 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:51:58] PROBLEM - WDQS high update lag on wdqs2005 is CRITICAL: 3660 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:52:04] RECOVERY - Apache HTTP on mw1248 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.673 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:52:06] PROBLEM - Apache HTTP on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:08] RECOVERY - PHP7 rendering on mw1345 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 9.540 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:10] RECOVERY - PHP7 rendering on mw1286 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.010 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:10] RECOVERY - Apache HTTP on mw1276 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.574 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:52:12] PROBLEM - Nginx local proxy to apache on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:14] PROBLEM - Nginx local proxy to apache on mw1257 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:16] PROBLEM - PHP7 rendering on mw1257 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:16] PROBLEM - Apache HTTP on mw1347 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:16] RECOVERY - PHP7 rendering on mw1244 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 4.728 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:20] PROBLEM - LVS HTTPS IPv4 #page on text-lb.codfw.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:52:20] PROBLEM - PHP7 rendering on mw1329 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:20] PROBLEM - PHP7 rendering on mw1254 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:24] PROBLEM - WDQS high update lag on wdqs2004 is CRITICAL: 3683 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:52:25] PROBLEM - PHP7 rendering on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:28] PROBLEM - Nginx local proxy to apache on mw1284 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:38] PROBLEM - WDQS high update lag on wdqs2006 is CRITICAL: 3693 ge 3600 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:52:42] PROBLEM - PHP7 rendering on mw1288 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:44] RECOVERY - PHP7 rendering on mw1339 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.897 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:50] PROBLEM - Nginx local proxy to apache on mw1314 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:54] PROBLEM - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:52:54] PROBLEM - Nginx local proxy to apache on mw1226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:54] PROBLEM - PHP7 rendering on mw1226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:52:55] PROBLEM - Nginx local proxy to apache on mw1232 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:52:58] RECOVERY - Apache HTTP on mw1339 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.728 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:52:58] PROBLEM - Nginx local proxy to apache on mw1276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:02] PROBLEM - Nginx local proxy to apache on mw1325 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:04] RECOVERY - Nginx local proxy to apache on mw1286 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.045 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:08] PROBLEM - Nginx local proxy to apache on mw1340 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:10] RECOVERY - Apache HTTP on mw1348 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.469 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:10] PROBLEM - Apache HTTP on mw1283 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:10] RECOVERY - PHP7 rendering on mw1230 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.881 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:53:12] RECOVERY - PHP7 rendering on mw1348 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.180 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:53:12] RECOVERY - Apache HTTP on mw1228 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.389 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:12] PROBLEM - Apache HTTP on mw1290 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:16] RECOVERY - Apache HTTP on mw1244 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.355 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:16] RECOVERY - Apache HTTP on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.785 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:20] PROBLEM - Nginx local proxy to apache on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:20] RECOVERY - Nginx local proxy to apache on mw1225 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.350 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:22] RECOVERY - Apache HTTP on mw1225 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.411 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:53:24] RECOVERY - LVS HTTPS IPv4 #page on text-lb.eqsin.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15185 bytes in 8.457 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:53:32] PROBLEM - Nginx local proxy to apache on mw1244 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:34] RECOVERY - PHP7 rendering on mw1276 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.921 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:53:40] PROBLEM - LVS HTTP IPv4 #page on appservers.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:53:42] PROBLEM - Apache HTTP on mw1314 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:48] RECOVERY - PHP7 rendering on mw1315 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.532 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:53:48] PROBLEM - Apache HTTP on mw1319 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:53:52] PROBLEM - Nginx local proxy to apache on mw1347 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:02] PROBLEM - Apache HTTP on mw1227 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:10] PROBLEM - Apache HTTP on mw1316 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:10] RECOVERY - Nginx local proxy to apache on mw1312 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.899 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:12] RECOVERY - Nginx local proxy to apache on mw1257 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.983 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:14] RECOVERY - PHP7 rendering on mw1257 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.178 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:14] RECOVERY - Apache HTTP on mw1281 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.689 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:14] RECOVERY - Apache HTTP on mw1284 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.095 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:16] PROBLEM - PHP7 rendering on mw1258 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:18] PROBLEM - Apache HTTP on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:18] RECOVERY - PHP7 rendering on mw1341 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 4.246 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:20] RECOVERY - PHP7 rendering on mw1254 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.622 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:20] PROBLEM - Nginx local proxy to apache on mw1243 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:24] PROBLEM - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:54:24] PROBLEM - PHP7 rendering on mw1243 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:25] PROBLEM - PHP7 rendering on mw1313 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:25] RECOVERY - PHP7 rendering on mw1277 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.133 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:25] RECOVERY - PHP7 rendering on mw1227 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.973 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:25] RECOVERY - Nginx local proxy to apache on mw1287 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.653 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:26] PROBLEM - Apache HTTP on mw1243 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:26] PROBLEM - Apache HTTP on mw1257 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:26] PROBLEM - Nginx local proxy to apache on mw1258 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:26] PROBLEM - Nginx local proxy to apache on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:27] PROBLEM - PHP7 rendering on mw1222 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:32] PROBLEM - Apache HTTP on mw1279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:42] RECOVERY - PHP7 rendering on mw1312 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.326 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:44] PROBLEM - LVS HTTPS IPv6 #page on text-lb.eqsin.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:54:46] PROBLEM - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:54:46] RECOVERY - Nginx local proxy to apache on mw1348 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.089 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:47] PROBLEM - PHP7 rendering on mw1229 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:54:47] RECOVERY - Nginx local proxy to apache on mw1317 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.735 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:47] PROBLEM - Apache HTTP on mw1346 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:47] PROBLEM - Nginx local proxy to apache on mw1241 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:52] PROBLEM - Nginx local proxy to apache on mw1345 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:54:58] RECOVERY - Nginx local proxy to apache on mw1226 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.279 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:54:58] RECOVERY - PHP7 rendering on mw1226 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.342 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:00] RECOVERY - PHP7 rendering on mw1287 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.994 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:02] PROBLEM - Nginx local proxy to apache on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:06] PROBLEM - Apache HTTP on mw1331 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:06] PROBLEM - Nginx local proxy to apache on mw1331 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:10] RECOVERY - Apache HTTP on mw1282 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.250 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:10] PROBLEM - Nginx local proxy to apache on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:12] RECOVERY - Nginx local proxy to apache on mw1228 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.552 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:12] PROBLEM - PHP7 rendering on mw1321 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:14] RECOVERY - Apache HTTP on mw1290 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.743 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:15] RECOVERY - Nginx local proxy to apache on mw1238 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.903 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:16] PROBLEM - Apache HTTP on mw1221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:18] PROBLEM - Apache HTTP on mw1251 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:18] PROBLEM - Nginx local proxy to apache on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:20] RECOVERY - PHP7 rendering on mw1228 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.181 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:20] RECOVERY - Nginx local proxy to apache on mw1221 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.679 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:24] PROBLEM - Nginx local proxy to apache on mw1247 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:24] RECOVERY - Nginx local proxy to apache on mw1233 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.970 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:26] RECOVERY - PHP7 rendering on mw1285 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.893 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:32] RECOVERY - Nginx local proxy to apache on mw1244 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.689 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:34] RECOVERY - Apache HTTP on mw1285 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.618 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:36] RECOVERY - LVS HTTP IPv4 #page on appservers.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 14631 bytes in 6.308 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:55:38] RECOVERY - Apache HTTP on mw1234 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.096 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:40] PROBLEM - PHP7 rendering on mw1333 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:40] RECOVERY - Apache HTTP on mw1314 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.395 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:44] PROBLEM - LVS HTTPS IPv4 #page on text-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:55:48] RECOVERY - Nginx local proxy to apache on mw1224 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.944 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:50] PROBLEM - Nginx local proxy to apache on mw1313 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:55:52] RECOVERY - Nginx local proxy to apache on mw1347 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.539 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:55:52] PROBLEM - PHP7 rendering on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:52] PROBLEM - PHP7 rendering on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:55:52] PROBLEM - PHP7 rendering on mw1347 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:00] RECOVERY - LVS HTTPS IPv4 #page on text-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15186 bytes in 9.106 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:56:00] RECOVERY - Nginx local proxy to apache on mw1285 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.870 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:00] PROBLEM - Apache HTTP on mw1332 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:56:02] PROBLEM - Apache HTTP on mw1297 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:56:02] PROBLEM - PHP7 rendering on mw1317 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:02] RECOVERY - Nginx local proxy to apache on mw1282 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.126 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:08] RECOVERY - Apache HTTP on mw1329 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.821 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:08] RECOVERY - Apache HTTP on mw1277 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.956 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:12] RECOVERY - Apache HTTP on mw1316 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.363 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:14] RECOVERY - Apache HTTP on mw1340 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.126 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:14] RECOVERY - Apache HTTP on mw1317 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.221 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:14] PROBLEM - Apache HTTP on mw1226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:56:18] PROBLEM - Nginx local proxy to apache on mw1346 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:56:18] RECOVERY - Nginx local proxy to apache on mw1243 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 5.064 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:20] RECOVERY - PHP7 rendering on mw1243 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 4.669 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:23] PROBLEM - PHP7 rendering on mw1283 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:24] RECOVERY - LVS HTTPS IPv4 #page on text-lb.ulsfo.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15185 bytes in 6.278 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:56:24] PROBLEM - Apache HTTP on mw1280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:56:25] PROBLEM - PHP7 rendering on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:28] RECOVERY - Apache HTTP on mw1243 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.306 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:30] RECOVERY - Nginx local proxy to apache on mw1341 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.424 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:32] PROBLEM - PHP7 rendering on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:36] RECOVERY - Apache HTTP on mw1279 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.459 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:40] PROBLEM - PHP7 rendering on mw1289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:46] RECOVERY - PHP7 rendering on mw1229 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.442 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:56:46] RECOVERY - Apache HTTP on mw1223 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.785 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:48] RECOVERY - Nginx local proxy to apache on mw1241 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.478 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:50] RECOVERY - Apache HTTP on mw1328 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.183 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:52] RECOVERY - Apache HTTP on mw1346 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.766 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:52] RECOVERY - Nginx local proxy to apache on mw1345 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.594 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:56:56] RECOVERY - Nginx local proxy to apache on mw1232 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.166 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:00] RECOVERY - PHP7 rendering on mw1332 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.041 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:00] PROBLEM - PHP7 rendering on mw1314 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:02] PROBLEM - PHP7 rendering on mw1238 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:04] PROBLEM - Nginx local proxy to apache on mw1279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:06] PROBLEM - Nginx local proxy to apache on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:10] PROBLEM - PHP7 rendering on mw1286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:10] PROBLEM - Apache HTTP on mw1276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:10] RECOVERY - Nginx local proxy to apache on mw1289 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.666 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:14] PROBLEM - Nginx local proxy to apache on mw1249 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:16] PROBLEM - PHP7 rendering on mw1323 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:20] RECOVERY - Nginx local proxy to apache on mw1247 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 5.614 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:22] RECOVERY - PHP7 rendering on mw1297 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.861 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:24] RECOVERY - Nginx local proxy to apache on mw1290 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.440 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:24] PROBLEM - Apache HTTP on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:34] RECOVERY - Apache HTTP on mw1323 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.151 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:38] RECOVERY - PHP7 rendering on mw1234 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.870 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:40] RECOVERY - LVS HTTPS IPv4 #page on text-lb.eqiad.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15184 bytes in 7.954 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:57:40] RECOVERY - PHP7 rendering on mw1333 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.152 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:42] PROBLEM - PHP7 rendering on mw1339 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:46] RECOVERY - Apache HTTP on mw1231 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.612 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:46] RECOVERY - Nginx local proxy to apache on mw1231 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.624 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:46] RECOVERY - Apache HTTP on mw1319 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.719 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:48] RECOVERY - PHP7 rendering on mw1280 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.022 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:48] RECOVERY - PHP7 rendering on mw1347 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.170 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:57:56] RECOVERY - Apache HTTP on mw1233 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.482 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:56] RECOVERY - Apache HTTP on mw1332 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.257 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:57:58] PROBLEM - Nginx local proxy to apache on mw1246 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:58] PROBLEM - Nginx local proxy to apache on mw1251 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:57:58] RECOVERY - Apache HTTP on mw1297 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.303 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:00] RECOVERY - PHP7 rendering on mw1317 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.872 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:02] RECOVERY - Apache HTTP on mw1343 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.852 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:04] RECOVERY - PHP7 rendering on mw1328 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.991 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:04] PROBLEM - PHP7 rendering on mw1251 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:05] PROBLEM - Apache HTTP on mw1239 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:05] PROBLEM - PHP7 rendering on mw1239 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:08] PROBLEM - Apache HTTP on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:10] PROBLEM - PHP7 rendering on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:12] RECOVERY - Nginx local proxy to apache on mw1346 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.322 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:17] RECOVERY - LVS HTTPS IPv4 #page on appservers.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 14623 bytes in 7.555 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:58:17] PROBLEM - Apache HTTP on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:17] RECOVERY - Apache HTTP on mw1347 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.930 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:17] PROBLEM - Apache HTTP on mw1288 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:18] RECOVERY - Apache HTTP on mw1280 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.160 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:20] RECOVERY - PHP7 rendering on mw1224 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.871 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:20] PROBLEM - Nginx local proxy to apache on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:20] RECOVERY - Apache HTTP on mw1257 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.015 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:22] RECOVERY - PHP7 rendering on mw1313 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 9.123 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:22] PROBLEM - Apache HTTP on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:34] RECOVERY - PHP7 rendering on mw1289 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.662 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:35] RECOVERY - Nginx local proxy to apache on mw1223 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.159 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:40] RECOVERY - LVS HTTPS IPv6 #page on text-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 6.942 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:58:40] RECOVERY - PHP7 rendering on mw1288 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.345 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:44] PROBLEM - Apache HTTP on mw1341 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:44] PROBLEM - PHP7 rendering on mw1316 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:52] RECOVERY - PHP7 rendering on mw1233 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.637 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:52] PROBLEM - PHP7 rendering on mw1256 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:52] PROBLEM - Nginx local proxy to apache on mw1235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:58:56] RECOVERY - PHP7 rendering on mw1238 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 6.355 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:56] RECOVERY - PHP7 rendering on mw1314 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.254 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:58:56] RECOVERY - Nginx local proxy to apache on mw1280 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.437 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:58:58] RECOVERY - Nginx local proxy to apache on mw1276 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.302 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:00] RECOVERY - Nginx local proxy to apache on mw1325 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 5.241 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:02] RECOVERY - Nginx local proxy to apache on mw1323 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.929 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:04] RECOVERY - Apache HTTP on mw1276 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.222 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:04] PROBLEM - PHP7 rendering on mw1345 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:06] RECOVERY - PHP7 rendering on mw1286 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.985 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:08] RECOVERY - Nginx local proxy to apache on mw1340 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.829 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:08] RECOVERY - PHP7 rendering on mw1323 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 5.950 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:08] PROBLEM - PHP7 rendering on mw1346 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:10] PROBLEM - Nginx local proxy to apache on mw1234 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:12] PROBLEM - Nginx local proxy to apache on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:14] RECOVERY - Apache HTTP on mw1251 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.607 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:16] PROBLEM - Apache HTTP on mw1284 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:22] RECOVERY - LVS HTTPS IPv6 #page on text-lb.ulsfo.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 7.847 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:59:23] PROBLEM - PHP7 rendering on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:23] PROBLEM - PHP7 rendering on mw1227 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:24] PROBLEM - LVS HTTPS IPv4 #page on text-lb.eqsin.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:59:25] PROBLEM - Nginx local proxy to apache on mw1287 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:28] PROBLEM - Apache HTTP on mw1278 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:28] PROBLEM - Nginx local proxy to apache on mw1227 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:44] RECOVERY - Nginx local proxy to apache on mw1283 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.012 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:46] PROBLEM - Nginx local proxy to apache on mw1348 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:54] RECOVERY - Apache HTTP on mw1224 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.336 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:54] PROBLEM - PHP7 rendering on mw1252 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:58] PROBLEM - LVS HTTP IPv4 #page on api.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [20:59:58] PROBLEM - PHP7 rendering on mw1226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [20:59:58] RECOVERY - Nginx local proxy to apache on mw1246 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.907 second response time https://wikitech.wikimedia.org/wiki/Application_servers [20:59:58] PROBLEM - Apache HTTP on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [20:59:58] PROBLEM - PHP7 rendering on mw1287 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:04] RECOVERY - PHP7 rendering on mw1251 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.012 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:04] RECOVERY - Apache HTTP on mw1348 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 3.583 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:05] RECOVERY - Apache HTTP on mw1239 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.930 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:06] RECOVERY - PHP7 rendering on mw1348 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 3.800 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:06] PROBLEM - PHP7 rendering on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:06] PROBLEM - PHP7 rendering on mw1324 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:10] PROBLEM - Nginx local proxy to apache on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:00:16] RECOVERY - LVS HTTPS IPv4 #page on text-lb.codfw.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15184 bytes in 6.193 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:00:17] RECOVERY - Apache HTTP on mw1288 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 9.711 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:17] PROBLEM - Nginx local proxy to apache on mw1297 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:00:18] RECOVERY - Nginx local proxy to apache on mw1332 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.785 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:20] PROBLEM - Apache HTTP on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:00:20] RECOVERY - Nginx local proxy to apache on mw1225 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.468 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:22] RECOVERY - PHP7 rendering on mw1343 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.632 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:22] PROBLEM - Nginx local proxy to apache on mw1221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:00:22] PROBLEM - PHP7 rendering on mw1228 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:22] RECOVERY - Apache HTTP on mw1225 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.573 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:24] PROBLEM - Nginx local proxy to apache on mw1233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:00:28] RECOVERY - Nginx local proxy to apache on mw1321 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.731 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:44] RECOVERY - Apache HTTP on mw1341 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.329 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:44] PROBLEM - PHP7 rendering on mw1281 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:46] RECOVERY - PHP7 rendering on mw1316 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.881 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:00:48] RECOVERY - Nginx local proxy to apache on mw1314 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.818 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:00:50] RECOVERY - LVS HTTPS IPv6 #page on text-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 4.648 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:00:52] RECOVERY - PHP7 rendering on mw1256 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.015 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:02] RECOVERY - Nginx local proxy to apache on mw1271 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 9.594 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:02] RECOVERY - Apache HTTP on mw1326 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 4.649 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:04] RECOVERY - Apache HTTP on mw1331 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.706 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:04] RECOVERY - Nginx local proxy to apache on mw1331 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.721 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:06] RECOVERY - PHP7 rendering on mw1345 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.764 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:06] RECOVERY - Nginx local proxy to apache on mw1326 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 4.784 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:06] PROBLEM - Apache HTTP on mw1277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:01:08] RECOVERY - PHP7 rendering on mw1321 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 5.505 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:10] RECOVERY - LVS HTTPS IPv6 #page on text-lb.codfw.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 5.273 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:01:11] RECOVERY - Apache HTTP on mw1283 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.436 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:11] RECOVERY - PHP7 rendering on mw1346 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.876 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:11] RECOVERY - Nginx local proxy to apache on mw1234 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.147 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:11] RECOVERY - Nginx local proxy to apache on mw1312 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.233 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:12] RECOVERY - Nginx local proxy to apache on mw1249 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 8.464 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:12] PROBLEM - Apache HTTP on mw1317 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers [21:01:14] RECOVERY - Apache HTTP on mw1221 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.881 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:14] RECOVERY - PHP7 rendering on mw1320 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 9.944 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:15] RECOVERY - Nginx local proxy to apache on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.390 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:16] RECOVERY - Apache HTTP on mw1284 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 7.557 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:16] RECOVERY - Apache HTTP on mw1320 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 8.619 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:18] RECOVERY - Apache HTTP on mw1289 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 6.357 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:20] RECOVERY - Nginx local proxy to apache on mw1229 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.248 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:20] RECOVERY - Nginx local proxy to apache on mw1342 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.433 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:20] RECOVERY - Apache HTTP on mw1235 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 4.153 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:22] RECOVERY - PHP7 rendering on mw1221 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.444 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:24] RECOVERY - Nginx local proxy to apache on mw1329 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 5.731 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:26] RECOVERY - LVS HTTPS IPv4 #page on text-lb.eqsin.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15185 bytes in 6.077 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:01:26] RECOVERY - PHP7 rendering on mw1277 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.333 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:27] RECOVERY - PHP7 rendering on mw1322 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 6.625 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:27] RECOVERY - PHP7 rendering on mw1279 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 7.334 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:27] RECOVERY - PHP7 rendering on mw1227 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 8.471 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:27] RECOVERY - Nginx local proxy to apache on mw1287 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 7.545 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:27] RECOVERY - Apache HTTP on mw1278 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 4.897 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:29] RECOVERY - Apache HTTP on mw1275 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.375 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:31] RECOVERY - Nginx local proxy to apache on mw1227 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 6.807 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:35] RECOVERY - PHP7 rendering on mw1282 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 4.948 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:37] RECOVERY - graphoid endpoints health on scb2003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [21:01:41] RECOVERY - Apache HTTP on mw1273 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.038 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:41] RECOVERY - Nginx local proxy to apache on mw1272 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.062 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:41] RECOVERY - Nginx local proxy to apache on mw1348 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 1.474 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:41] RECOVERY - PHP7 rendering on mw1339 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 3.347 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:43] RECOVERY - Nginx local proxy to apache on mw1313 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 629 bytes in 2.246 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:43] RECOVERY - PHP7 rendering on mw1235 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.174 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:43] RECOVERY - mobileapps endpoints health on scb1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [21:01:47] RECOVERY - PHP7 rendering on mw1270 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.145 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:47] RECOVERY - Apache HTTP on mw1261 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.451 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:49] RECOVERY - graphoid endpoints health on scb1003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [21:01:49] RECOVERY - PHP7 rendering on mw1252 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.170 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:49] RECOVERY - PHP7 rendering on mw1226 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.119 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:53] RECOVERY - LVS HTTP IPv4 #page on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 23765 bytes in 1.557 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:01:53] RECOVERY - Nginx local proxy to apache on mw1265 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.046 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:53] RECOVERY - Nginx local proxy to apache on mw1251 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.044 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:53] RECOVERY - graphoid endpoints health on scb1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [21:01:53] RECOVERY - restbase endpoints health on restbase2012 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:01:55] RECOVERY - PHP7 rendering on mw1287 is OK: HTTP OK: HTTP/1.1 200 OK - 76235 bytes in 1.431 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:55] RECOVERY - Apache HTTP on mw1342 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 1.431 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:59] RECOVERY - Graphoid LVS codfw on graphoid.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Graphoid [21:01:59] RECOVERY - Nginx local proxy to apache on mw1262 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.044 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:59] RECOVERY - Apache HTTP on mw1249 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.048 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:01:59] RECOVERY - PHP7 rendering on mw1271 is OK: HTTP OK: HTTP/1.1 200 OK - 76233 bytes in 0.116 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:59] RECOVERY - PHP7 rendering on mw1223 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.119 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:01:59] RECOVERY - Apache HTTP on mw1227 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.491 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:01] RECOVERY - PHP7 rendering on mw1239 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.120 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:01] RECOVERY - restbase endpoints health on restbase2022 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:01] RECOVERY - PHP7 rendering on mw1268 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.139 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:01] RECOVERY - PHP7 rendering on mw1225 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.146 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:01] RECOVERY - PHP7 rendering on mw1324 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.154 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:03] RECOVERY - Restbase LVS codfw on restbase.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [21:02:03] RECOVERY - graphoid endpoints health on scb1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [21:02:03] RECOVERY - Nginx local proxy to apache on mw1277 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 628 bytes in 0.746 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:03] RECOVERY - PHP7 rendering on mw1325 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.137 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:05] RECOVERY - graphoid endpoints health on scb2005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/graphoid [21:02:05] RECOVERY - mobileapps endpoints health on scb1003 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/mobileapps [21:02:07] RECOVERY - restbase endpoints health on restbase2013 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:07] RECOVERY - restbase endpoints health on restbase2015 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:07] RECOVERY - restbase endpoints health on restbase2019 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:07] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Citoid [21:02:07] RECOVERY - Nginx local proxy to apache on mw1273 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 627 bytes in 0.045 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:07] RECOVERY - Apache HTTP on mw1274 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.033 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:08] RECOVERY - Apache HTTP on mw1263 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.043 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:08] RECOVERY - Apache HTTP on mw1265 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.040 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:09] RECOVERY - Apache HTTP on mw1325 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.041 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:09] RECOVERY - Apache HTTP on mw1269 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.039 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:10] RECOVERY - Apache HTTP on mw1226 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 626 bytes in 0.033 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:02:10] RECOVERY - PHP7 rendering on mw1269 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.140 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:11] RECOVERY - PHP7 rendering on mw1263 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.136 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:11] RECOVERY - PHP7 rendering on mw1275 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.124 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:12] RECOVERY - restbase endpoints health on restbase2023 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:12] RECOVERY - PHP7 rendering on mw1273 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.154 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:13] RECOVERY - PHP7 rendering on mw1274 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.147 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:13] RECOVERY - restbase endpoints health on restbase2016 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:14] RECOVERY - PHP7 rendering on mw1331 is OK: HTTP OK: HTTP/1.1 200 OK - 76234 bytes in 0.178 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:02:14] RECOVERY - restbase endpoints health on restbase2017 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:02:33] !log ats-tls-restart on cp3064 [21:02:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:03:45] RECOVERY - WDQS high update lag on wdqs1003 is OK: (C)3600 ge (W)1200 ge 98.1 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [21:03:45] RECOVERY - restbase endpoints health on restbase1026 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:45] RECOVERY - restbase endpoints health on restbase1019 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:45] RECOVERY - restbase endpoints health on restbase2018 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:45] RECOVERY - restbase endpoints health on restbase1023 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:49] RECOVERY - restbase endpoints health on restbase1018 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:49] RECOVERY - restbase endpoints health on restbase1016 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:49] RECOVERY - restbase endpoints health on restbase-dev1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:49] RECOVERY - restbase endpoints health on restbase1017 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:03:53] RECOVERY - restbase endpoints health on restbase-dev1006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:04:01] RECOVERY - WDQS high update lag on wdqs2005 is OK: (C)3600 ge (W)1200 ge 44.47 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [21:04:01] RECOVERY - PyBal IPVS diff check on lvs1016 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [21:04:05] RECOVERY - recommendation_api endpoints health on scb1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [21:04:07] RECOVERY - restbase endpoints health on restbase-dev1005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/restbase [21:04:07] RECOVERY - Restbase edge codfw on text-lb.codfw.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [21:04:07] RECOVERY - recommendation_api endpoints health on scb2002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/recommendation_api [21:04:11] RECOVERY - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:04:23] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [21:04:25] RECOVERY - High average POST latency for mw requests on api_appserver in eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=api_appserver&var-method=POST [21:04:25] RECOVERY - High average POST latency for mw requests on appserver in eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=appserver&var-method=POST [21:04:29] RECOVERY - WDQS high update lag on wdqs2004 is OK: (C)3600 ge (W)1200 ge 15.99 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [21:04:31] RECOVERY - wikidata.org dispatch lag is REALLY high ---4000s- on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1915 bytes in 0.071 second response time https://phabricator.wikimedia.org/project/view/71/ [21:04:41] RECOVERY - WDQS high update lag on wdqs2006 is OK: (C)3600 ge (W)1200 ge 52 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [21:04:51] RECOVERY - Cxserver LVS codfw on cxserver.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/CX [21:04:51] RECOVERY - High average GET latency for mw requests on appserver in eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=appserver&var-method=GET [21:05:11] RECOVERY - Restbase edge ulsfo on text-lb.ulsfo.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [21:05:21] RECOVERY - High average GET latency for mw requests on api_appserver in eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=api_appserver&var-method=GET [21:06:21] (03PS1) 10Krinkle: Lower wgHTTPTimeout from default 25s to 0.5s [mediawiki-config] - 10https://gerrit.wikimedia.org/r/567365 [21:06:43] RECOVERY - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/CX [21:06:49] RECOVERY - Restbase edge esams on text-lb.esams.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [21:07:07] PROBLEM - MediaWiki memcached error rate on icinga1001 is CRITICAL: 8228 gt 5000 https://wikitech.wikimedia.org/wiki/Memcached https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=1&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [21:07:21] RECOVERY - Logstash Elasticsearch indexing errors on icinga1001 is OK: (C)8 ge (W)1 ge 0.04583 https://wikitech.wikimedia.org/wiki/Logstash%23Indexing_errors https://logstash.wikimedia.org/goto/1cee1f1b5d4e6c5e06edb3353a2a4b83 https://grafana.wikimedia.org/dashboard/db/logstash [21:07:33] RECOVERY - PyBal IPVS diff check on lvs1015 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [21:07:54] (03CR) 10Jforrester: [C: 03+1] "Seems reasonable." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/567365 (owner: 10Krinkle) [21:07:57] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1002.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:08:07] RECOVERY - Restbase edge eqiad on text-lb.eqiad.wikimedia.org is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [21:09:43] PROBLEM - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is CRITICAL: /v4/marker/pin-m-fuel+ffffff@2x.png (scaled pushpin marker with an icon) timed out before a response was received https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:10:47] PROBLEM - MediaWiki memcached error rate on icinga1001 is CRITICAL: 2.001e+04 gt 5000 https://wikitech.wikimedia.org/wiki/Memcached https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=1&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [21:12:07] PROBLEM - High average GET latency for mw requests on appserver in eqiad on icinga1001 is CRITICAL: cluster=appserver code={200,204} handler=proxy:unix:/run/php/fpm-www.sock https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=appserver&var-method= [21:13:23] RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:14:23] RECOVERY - MediaWiki memcached error rate on icinga1001 is OK: (C)5000 gt (W)1000 gt 14 https://wikitech.wikimedia.org/wiki/Memcached https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=1&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [21:15:15] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1002.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:15:45] RECOVERY - High average GET latency for mw requests on appserver in eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=appserver&var-method=GET [21:18:49] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1002.eqiad.wmnet, maps1001.eqiad.wmnet, maps1004.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:19:19] !log restart varnish-fe and ats-tls on cp3056 [21:19:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:20:17] PROBLEM - kartotherian endpoints health on maps1002 is CRITICAL: /v4/marker/pin-m-fuel+ffffff.png (Untitled test) is CRITICAL: Could not fetch url http://10.64.16.42:6533/v4/marker/pin-m-fuel+ffffff.png: Generic connection error: HTTPConnectionPool(host=10.64.16.42, port=6533): Max retries exceeded with url: /v4/marker/pin-m-fuel+ffffff.png (Caused by NewConnectionError(urllib3.connection.HTTPConnection object at 0x7f6750e68f98: [21:20:17] sh a new connection: [Errno 111] Connection refused,)): /v4/marker/pin-m+ffffff@2x.png (Untitled test) is CRITICAL: Could not fetch url http://10.64.16.42:6533/v4/marker/pin-m+ffffff@2x.png: Generic connection error: HTTPConnectionPool(host=10.64.16.42, port=6533): Max retries exceeded with url: /v4/marker/pin-m+ffffff@2x.png (Caused by NewConnectionError(urllib3.connection.HTTPConnection object at 0x7f6750e68f28: Failed to estab [21:20:17] tion: [Errno 111] Connection refused,)): /v4/marker/pin-m-fuel+ffffff@2x.png (scaled pushpin marker with an icon) is CRITICAL: Could not fetch url http://10.64.16.42:6533/v4/ma https://wikitech.wikimedia.org/wiki/Services/Monitoring/kartotherian [21:20:37] RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:20:41] RECOVERY - PyBal backends health check on lvs1015 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:22:07] RECOVERY - kartotherian endpoints health on maps1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/kartotherian [21:22:29] PROBLEM - Host cp3051 is DOWN: PING CRITICAL - Packet loss = 100% [21:23:30] !log restart kartotherian on maps1002 [21:23:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:27:57] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1002.eqiad.wmnet, maps1001.eqiad.wmnet, maps1004.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:31:27] PROBLEM - Maps HTTPS on maps1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Maps/RunBook [21:33:11] RECOVERY - Maps HTTPS on maps1003 is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 4.826 second response time https://wikitech.wikimedia.org/wiki/Maps/RunBook [21:34:27] PROBLEM - Maps HTTPS on maps1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Maps/RunBook [21:34:29] PROBLEM - LVS HTTP IPv4 #page on kartotherian.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:35:13] RECOVERY - PyBal backends health check on lvs1015 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:36:13] RECOVERY - LVS HTTP IPv4 #page on kartotherian.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 2.737 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [21:36:13] RECOVERY - Maps HTTPS on maps1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 5.266 second response time https://wikitech.wikimedia.org/wiki/Maps/RunBook [21:38:22] !log powercycling cp3051 - T238305 [21:38:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:38:25] T238305: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 [21:38:59] RECOVERY - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:42:32] !log akosiaris@cumin1001 conftool action : set/pooled=no; selector: name=maps1003.* [21:42:33] RECOVERY - Host cp3051 is UP: PING OK - Packet loss = 0%, RTA = 83.33 ms [21:42:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:42:43] !log test depool maps1003 [21:42:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:44:21] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1002.eqiad.wmnet, maps1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:44:21] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1002.eqiad.wmnet, maps1001.eqiad.wmnet, maps1004.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:44:29] PROBLEM - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is CRITICAL: /v4/marker/pin-m-fuel+ffffff.png (Untitled test) timed out before a response was received https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:45:01] !log akosiaris@cumin1001 conftool action : set/pooled=yes; selector: name=maps1003.* [21:45:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:45:12] !log repool maps1003 [21:45:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:46:15] RECOVERY - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:47:49] PROBLEM - Maps HTTPS on maps1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Maps/RunBook [21:48:31] RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:48:45] RECOVERY - Maps HTTPS on maps1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 4.755 second response time https://wikitech.wikimedia.org/wiki/Maps/RunBook [21:49:43] PROBLEM - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is CRITICAL: /v4/marker/pin-m-fuel+ffffff.png (Untitled test) timed out before a response was received https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:51:47] RECOVERY - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:52:43] RECOVERY - PyBal backends health check on lvs1015 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:55:13] PROBLEM - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is CRITICAL: /v4/marker/pin-m-fuel+ffffff.png (Untitled test) timed out before a response was received: /v4/marker/pin-m-fuel+ffffff@2x.png (scaled pushpin marker with an icon) timed out before a response was received https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [21:56:15] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1002.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:58:45] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:04:09] RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:07:05] RECOVERY - PyBal backends health check on lvs1015 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:08:35] PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1004.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:09:41] PROBLEM - Maps HTTPS on maps1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Maps/RunBook [22:11:09] RECOVERY - Maps HTTPS on maps1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 1.652 second response time https://wikitech.wikimedia.org/wiki/Maps/RunBook [22:13:25] RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:14:37] PROBLEM - LVS HTTP IPv4 #page on kartotherian.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [22:15:09] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1002.eqiad.wmnet, maps1004.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:17:42] RECOVERY - LVS HTTP IPv4 #page on kartotherian.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 7.748 second response time https://wikitech.wikimedia.org/wiki/LVS%23Diagnosing_problems [22:21:20] RECOVERY - PyBal backends health check on lvs1015 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:27:18] PROBLEM - PyBal backends health check on lvs1015 is CRITICAL: PYBAL CRITICAL - CRITICAL - kartotherian-ssl_443: Servers maps1003.eqiad.wmnet, maps1004.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:29:56] RECOVERY - PyBal backends health check on lvs1015 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:48:34] RECOVERY - Kartotherian LVS eqiad #page on kartotherian.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Maps%23Kartotherian [23:03:00] PROBLEM - Maps HTTPS on maps1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Maps/RunBook [23:04:44] RECOVERY - Maps HTTPS on maps1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1286 bytes in 4.744 second response time https://wikitech.wikimedia.org/wiki/Maps/RunBook [23:05:14] 10Operations, 10Traffic, 10Performance Issue, 10Wikimedia-Incident: Time-out error; Babel/WikibaseRepo being somehow uncached, overloading the API, and causing general outage - https://phabricator.wikimedia.org/T243713 (10Jdforrester-WMF) [23:13:48] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [23:15:36] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [23:44:04] PROBLEM - HTTPS-blog on blog.wikimedia.org is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:SSL connect attempt failed https://phabricator.wikimedia.org/tag/wikimedia-blog/ [23:45:52] RECOVERY - HTTPS-blog on blog.wikimedia.org is OK: SSL OK - Certificate blog.wikimedia.org valid until 2020-03-05 05:48:16 +0000 (expires in 38 days) https://phabricator.wikimedia.org/tag/wikimedia-blog/