[00:00:44] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:01:36] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:02:28] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75587 bytes in 1.026 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:05:34] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:07:18] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75600 bytes in 0.394 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:10:28] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:12:08] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:12:18] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75602 bytes in 9.329 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:17:30] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75600 bytes in 0.376 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:31:44] RECOVERY - Too many messages in kafka logging-eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Logstash%23Kafka_consumer_lag https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&to=now&orgId=1&var-datasource=eqiad+prometheus/ops&var-cluster=logging-eqiad&var-topic=All&var-consumer_group=All [00:32:22] PROBLEM - SSH on db2082.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [00:34:30] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:35:26] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:36:18] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75588 bytes in 8.892 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:38:58] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75586 bytes in 0.632 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:47:38] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:50:22] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:50:54] RECOVERY - mediawiki originals uploads -hourly- for codfw on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Swift/How_To%23mediawiki_originals_uploads https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [00:51:02] RECOVERY - mediawiki originals uploads -hourly- for eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Swift/How_To%23mediawiki_originals_uploads https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [00:51:16] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75586 bytes in 0.847 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:53:58] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75586 bytes in 0.371 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:54:46] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:55:20] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:55:52] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:56:28] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75586 bytes in 0.393 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:57:00] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75586 bytes in 0.364 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [00:59:22] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 1.080 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:00:16] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:01:36] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:03:52] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75598 bytes in 1.353 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:04:12] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:05:10] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.780 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:05:26] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:05:52] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.842 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:06:06] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:07:28] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:07:48] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.600 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:08:58] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.669 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:09:58] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:10:40] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:10:56] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.641 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:12:20] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.162 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:13:46] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:14:26] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:15:08] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:15:26] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.881 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:16:00] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:16:06] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.599 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:16:50] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.566 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:17:10] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.357 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:17:40] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.167 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:17:43] hmm [01:21:10] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:22:50] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.823 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:26:24] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:28:32] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:29:06] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:29:54] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.418 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:30:48] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.366 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:33:04] RECOVERY - SSH on db2082.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [01:33:52] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75628 bytes in 0.555 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:34:12] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:34:36] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:35:40] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:36:20] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.777 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:37:20] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.161 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:37:26] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:37:46] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75596 bytes in 0.608 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:38:38] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:38:44] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:39:08] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75562 bytes in 0.636 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:40:16] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:40:18] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75562 bytes in 0.568 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:40:24] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75562 bytes in 0.355 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:41:56] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75529 bytes in 1.048 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:42:10] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:43:26] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:45:06] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:45:08] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:45:08] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.631 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:45:14] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:45:44] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.372 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:45:50] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:46:56] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.375 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:46:58] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75529 bytes in 8.710 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:48:40] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.368 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:49:06] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:49:24] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.639 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:50:30] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:50:34] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:53:58] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.580 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:54:04] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.865 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:54:10] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:55:52] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.671 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:56:18] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.387 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:56:34] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:58:18] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.842 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [01:59:56] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:00:56] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:02:34] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:02:38] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.815 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:03:02] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:03:30] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.151 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:04:16] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.581 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:04:44] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.621 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:06:50] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:07:28] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:08:22] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:09:10] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.404 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:10:04] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.363 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:10:12] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:10:58] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:12:42] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.795 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:13:46] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.827 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:14:04] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:14:06] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.416 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:14:56] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:15:44] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:15:46] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.833 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:16:38] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.806 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:17:26] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.779 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:19:22] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:20:36] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:20:42] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:21:30] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:22:34] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75529 bytes in 9.150 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:22:40] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:22:54] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.414 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:23:12] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.564 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:23:18] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:24:22] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75529 bytes in 1.056 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:26:08] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75529 bytes in 9.457 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:28:48] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:29:04] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:29:30] (03PS1) 10Andrew Bogott: Openstack buster packages: pin python3-ceph and related [puppet] - 10https://gerrit.wikimedia.org/r/593877 (https://phabricator.wikimedia.org/T251294) [02:29:34] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:29:50] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:30:32] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.376 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:30:47] (03PS2) 10Andrew Bogott: Openstack buster packages: pin python3-ceph and related [puppet] - 10https://gerrit.wikimedia.org/r/593877 (https://phabricator.wikimedia.org/T251294) [02:30:48] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:31:34] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.157 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:31:54] (03CR) 10Andrew Bogott: [C: 03+2] Openstack buster packages: pin python3-ceph and related [puppet] - 10https://gerrit.wikimedia.org/r/593877 (https://phabricator.wikimedia.org/T251294) (owner: 10Andrew Bogott) [02:32:20] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.420 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:32:38] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.594 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:33:52] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:35:02] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.392 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:35:24] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:35:28] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:36:24] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:36:24] !log volker-e@deploy1001 Started deploy [design/style-guide@f0d467b]: Deploy design/style-guide: [02:36:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:36:31] !log volker-e@deploy1001 Finished deploy [design/style-guide@f0d467b]: Deploy design/style-guide: (duration: 00m 07s) [02:36:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:37:14] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:37:18] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75529 bytes in 8.838 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:37:26] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.881 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:38:06] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.392 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:38:06] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.150 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:39:18] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:39:50] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:40:44] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.851 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:40:46] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75530 bytes in 1.138 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:42:12] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:42:48] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.445 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:45:34] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:45:36] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:47:20] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.566 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:48:58] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.364 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:49:28] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:52:18] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:53:04] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.354 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:53:16] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.629 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:53:50] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:53:58] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.345 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:56:02] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:57:52] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [02:59:34] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.391 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:00:16] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.431 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:00:34] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:00:54] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:01:10] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:01:22] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.629 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:02:14] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.854 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:02:26] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:02:50] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.638 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:04:16] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75530 bytes in 9.592 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:04:24] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.638 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:04:38] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.578 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:05:22] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:07:02] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:07:04] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.605 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:09:10] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:10:32] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.783 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:10:48] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.427 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:11:54] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:12:38] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:14:30] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:14:42] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:15:22] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:15:40] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:16:16] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:17:24] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75530 bytes in 1.089 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:18:04] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.583 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:18:04] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.821 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:18:14] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.190 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:18:58] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75530 bytes in 1.145 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:19:08] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.622 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:19:52] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.367 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:20:40] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:20:58] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:22:14] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:22:34] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:23:46] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:23:56] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.882 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:24:14] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.394 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:24:48] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:24:48] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:24:58] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:26:30] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75530 bytes in 1.090 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:26:32] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:26:40] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.604 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:27:20] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75530 bytes in 1.182 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:28:14] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.156 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:28:20] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.581 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:29:48] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.447 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:30:04] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.576 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:30:46] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:30:58] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:31:30] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:32:26] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:32:28] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.412 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:33:14] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.636 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:34:38] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:34:54] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:35:20] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:36:54] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:37:52] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:37:54] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:38:04] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:38:14] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75528 bytes in 0.149 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:38:36] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:39:44] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.412 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:40:04] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.637 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:40:28] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.802 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:40:46] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.834 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:41:24] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.150 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:41:28] PROBLEM - Too many messages in kafka logging-eqiad on icinga1001 is CRITICAL: cluster=misc exported_cluster=logging-eqiad group=logstash instance=kafkamon1001:9501 job=burrow partition={2,3} site=eqiad topic={udp_localhost-info,udp_localhost-warning} https://wikitech.wikimedia.org/wiki/Logstash%23Kafka_consumer_lag https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&to=now&orgId=1&var-datasource=eqiad+prometh [03:41:28] er=logging-eqiad&var-topic=All&var-consumer_group=All [03:41:38] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.617 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:42:10] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.858 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:42:12] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.822 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:43:18] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.811 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:46:28] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:46:46] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:47:00] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:47:34] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:48:08] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:48:38] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:48:44] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.834 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:48:52] RECOVERY - Too many messages in kafka logging-eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Logstash%23Kafka_consumer_lag https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&to=now&orgId=1&var-datasource=eqiad+prometheus/ops&var-cluster=logging-eqiad&var-topic=All&var-consumer_group=All [03:49:18] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.102 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:49:58] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:50:04] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.401 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:50:18] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:50:20] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.595 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:52:20] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 9.405 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:53:34] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:53:34] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:54:06] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:54:52] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:55:14] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.623 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:55:46] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.579 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:56:34] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.626 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:57:04] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.371 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:57:14] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.582 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:57:34] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:58:52] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [03:59:00] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.179 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:00:06] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:00:36] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.368 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:01:12] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.836 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:01:26] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:01:44] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:01:58] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:02:06] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:03:08] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.605 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:03:26] PROBLEM - Kafka Broker Under Replicated Partitions on kafka-jumbo1004 is CRITICAL: 10 ge 10 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1004 [04:03:40] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.420 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:03:48] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.583 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:05:26] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:05:32] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.030 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:07:04] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.384 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:07:18] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 9.981 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:07:22] PROBLEM - Too many messages in kafka logging-eqiad on icinga1001 is CRITICAL: cluster=misc exported_cluster=logging-eqiad group=logstash instance=kafkamon1001:9501 job=burrow partition={4,5} site=eqiad topic={udp_localhost-info,udp_localhost-warning} https://wikitech.wikimedia.org/wiki/Logstash%23Kafka_consumer_lag https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&to=now&orgId=1&var-datasource=eqiad+prometh [04:07:22] er=logging-eqiad&var-topic=All&var-consumer_group=All [04:07:54] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:08:32] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:09:50] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:10:12] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.596 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:10:32] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:12:14] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.448 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:12:26] PROBLEM - Kafka Broker Replica Max Lag on kafka-jumbo1006 is CRITICAL: 5.058e+06 ge 5e+06 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=16&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1006 [04:12:48] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:13:18] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.814 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:13:20] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:14:18] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.879 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:14:46] RECOVERY - Too many messages in kafka logging-eqiad on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Logstash%23Kafka_consumer_lag https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?from=now-3h&to=now&orgId=1&var-datasource=eqiad+prometheus/ops&var-cluster=logging-eqiad&var-topic=All&var-consumer_group=All [04:15:00] PROBLEM - Backup freshness on backup1001 is CRITICAL: All failures: 4 (miscweb1002, ...), Fresh: 89 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [04:15:04] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:15:04] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:15:16] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.836 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:15:44] PROBLEM - Kafka Broker Under Replicated Partitions on kafka-jumbo1002 is CRITICAL: 10 ge 10 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1002 [04:16:56] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.359 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:17:06] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:18:40] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.423 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:18:40] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.575 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:20:28] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:21:48] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:22:10] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.350 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:22:36] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.366 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:23:16] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:23:54] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.403 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:24:48] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:25:00] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.633 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:25:24] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.353 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:25:26] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:25:44] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:27:02] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:27:26] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:28:16] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:28:16] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:28:34] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:29:04] PROBLEM - Kafka Broker Under Replicated Partitions on kafka-jumbo1001 is CRITICAL: 10 ge 10 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1001 [04:29:16] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.867 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:29:58] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.239 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:30:12] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:30:14] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.416 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:31:38] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:31:54] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.380 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:32:52] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75547 bytes in 1.675 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:34:16] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.584 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:34:30] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.149 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:34:46] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:35:10] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.626 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:35:14] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:35:28] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75545 bytes in 0.844 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:35:56] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:36:42] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:36:54] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:37:38] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:39:16] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:39:20] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.614 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:39:58] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:40:08] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.588 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:40:14] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.911 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:40:18] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:40:26] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.600 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:41:18] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.031 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:41:58] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.046 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:43:44] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:44:50] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.365 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:44:56] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:45:02] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:45:16] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:46:04] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:46:18] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 9.419 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:46:48] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:47:14] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.569 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:48:52] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:50:12] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.600 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:50:20] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.062 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:50:54] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.839 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:52:04] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:52:18] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.299 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:52:24] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.042 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:52:50] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:53:46] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.357 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:54:24] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:54:34] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.797 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:55:14] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:55:48] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:56:16] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.841 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:57:06] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 2.914 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:57:10] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:57:16] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:57:44] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.139 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:58:38] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:58:44] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75498 bytes in 0.839 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:58:46] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:58:50] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75505 bytes in 1.130 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [04:59:48] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75499 bytes in 1.057 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:01:16] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:01:50] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:01:56] PROBLEM - MariaDB Slave Lag: s4 on db1125 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 386.47 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave [05:02:18] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75522 bytes in 0.371 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:02:34] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:04:02] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75537 bytes in 0.554 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:04:38] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:04:50] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75537 bytes in 0.798 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:04:56] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75539 bytes in 9.427 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:05:26] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:07:58] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.417 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:08:58] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.811 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:09:40] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:10:10] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:12:46] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:13:14] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 1.860 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:13:32] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:13:38] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:13:44] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.795 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:14:08] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:14:30] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.802 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:15:22] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.374 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:15:36] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.604 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:15:38] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.338 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:16:30] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.701 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:18:06] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:18:28] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:18:54] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:19:00] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.326 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:19:22] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:19:34] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.587 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:20:00] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:20:16] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.587 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:21:24] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.817 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:21:48] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.801 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:21:52] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.842 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:22:10] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:22:14] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:22:46] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:23:26] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:24:14] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.619 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:24:44] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:25:10] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:25:44] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.848 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:25:58] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.397 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:27:22] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.111 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:27:42] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:27:56] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75539 bytes in 2.023 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:28:24] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:29:22] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:29:38] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:30:04] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.670 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:30:30] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.020 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:30:58] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:31:42] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:32:08] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.595 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:32:12] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.368 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:34:12] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.410 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:36:08] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:36:10] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:36:40] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:38:16] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:38:20] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:39:00] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.038 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:39:18] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.384 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:42:14] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.887 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:43:22] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:43:30] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.129 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:43:40] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:44:26] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.398 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:46:48] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:48:06] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:48:42] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:49:08] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:50:16] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.080 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:50:34] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:51:08] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:52:18] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.065 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:54:06] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.479 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:54:22] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.587 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:55:08] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:55:52] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.824 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:56:12] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:56:50] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.067 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:57:06] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:58:16] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.039 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:58:26] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.301 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:58:56] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:59:12] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [05:59:18] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.997 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:00:00] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.060 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:00:42] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:01:38] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75540 bytes in 1.992 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:02:30] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.419 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:03:02] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:03:12] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:03:30] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:04:06] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:04:18] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.847 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:04:34] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.806 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:04:50] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:04:54] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.581 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:05:12] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.884 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:05:16] (03PS5) 10DannyS712: Remove "Create a book" link on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561403 (https://phabricator.wikimedia.org/T241683) [06:06:26] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.571 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:06:28] PROBLEM - PHP7 rendering on mw1361 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:07:20] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:09:32] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.821 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:09:46] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:09:48] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75538 bytes in 0.853 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:10:04] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:11:12] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:11:12] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:11:47] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75494 bytes in 1.224 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:12:02] PROBLEM - PHP7 rendering on mw1384 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:12:04] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.646 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:12:10] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.627 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:12:44] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.824 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:12:54] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.369 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:13:20] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.626 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:13:46] RECOVERY - PHP7 rendering on mw1384 is OK: HTTP OK: HTTP/1.1 200 OK - 75493 bytes in 1.979 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:14:20] PROBLEM - PHP7 rendering on mw1352 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:14:38] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:15:40] PROBLEM - PHP7 rendering on mw1355 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:16:38] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:17:46] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:18:12] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:18:44] PROBLEM - PHP7 rendering on mw1357 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:18:50] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:19:22] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:19:50] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.623 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:20:00] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.368 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:20:14] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.595 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:20:32] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.798 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:23:20] PROBLEM - rsyslog in eqiad is failing to deliver messages on icinga1001 is CRITICAL: action=fwd_centrallog2001.codfw.wmnet:6514 https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=eqiad+prometheus/ops [06:24:44] PROBLEM - PHP7 rendering on mw1350 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:24:52] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:25:06] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:25:12] RECOVERY - rsyslog in eqiad is failing to deliver messages on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=eqiad+prometheus/ops [06:25:24] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:25:48] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75494 bytes in 2.750 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:26:42] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75494 bytes in 1.805 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:28:44] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.369 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:29:08] PROBLEM - ores uWSGI web app on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/Services/ores [06:29:14] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=atlas_exporter site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [06:29:26] PROBLEM - Check size of conntrack table on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack [06:29:50] PROBLEM - puppet last run on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:30:12] PROBLEM - Check systemd state on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:30:34] PROBLEM - PHP7 rendering on mw1359 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:31:02] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [06:31:24] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [06:31:28] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:32:10] PROBLEM - MD RAID on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [06:33:12] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75494 bytes in 1.562 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:33:34] PROBLEM - PHP7 rendering on mw1351 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:33:56] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75494 bytes in 2.312 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:34:24] PROBLEM - PHP7 rendering on mw1384 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:38:04] PROBLEM - PHP7 rendering on mw1356 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:38:50] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:39:14] PROBLEM - Number of backend failures per minute from CirrusSearch on graphite1004 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [600.0] https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=1&var-cluster=eqiad&var-smoothing=1&panelId=9&fullscreen [06:40:02] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.418 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:41:20] PROBLEM - configured eth on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/check_eth [06:41:48] RECOVERY - PHP7 rendering on mw1384 is OK: HTTP OK: HTTP/1.1 200 OK - 75493 bytes in 8.280 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:43:20] PROBLEM - ores_workers_running on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/ORES [06:43:52] PROBLEM - rsyslog in codfw is failing to deliver messages on icinga1001 is CRITICAL: action=fwd_centrallog2001.codfw.wmnet:6514 https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=codfw+prometheus/ops [06:44:26] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [06:44:52] PROBLEM - PHP7 rendering on mw1349 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:46:04] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.639 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:46:38] RECOVERY - Number of backend failures per minute from CirrusSearch on graphite1004 is OK: OK: Less than 20.00% above the threshold [300.0] https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=1&var-cluster=eqiad&var-smoothing=1&panelId=9&fullscreen [06:48:12] PROBLEM - Check whether ferm is active by checking the default input chain on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [06:48:48] PROBLEM - Disk space on ores1003 is CRITICAL: connect to address 10.64.16.94 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=ores1003&var-datasource=eqiad+prometheus/ops [06:48:52] PROBLEM - MediaWiki memcached error rate on icinga1001 is CRITICAL: 6151 gt 5000 https://wikitech.wikimedia.org/wiki/Memcached https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=1&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [06:49:42] RECOVERY - Check size of conntrack table on ores1003 is OK: OK: nf_conntrack is 0 % full https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack [06:50:12] PROBLEM - PHP7 rendering on mw1384 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:50:30] RECOVERY - Check systemd state on ores1003 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:50:52] PROBLEM - PHP7 rendering on mw1354 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:50:54] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.375 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:52:00] RECOVERY - PHP7 rendering on mw1384 is OK: HTTP OK: HTTP/1.1 200 OK - 75493 bytes in 4.935 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:52:12] PROBLEM - PHP7 rendering on mw1383 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:52:34] RECOVERY - ores_workers_running on ores1003 is OK: PROCS OK: 91 processes with command name celery https://wikitech.wikimedia.org/wiki/ORES [06:53:14] RECOVERY - puppet last run on ores1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [06:53:52] RECOVERY - MD RAID on ores1003 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [06:53:56] RECOVERY - PHP7 rendering on mw1383 is OK: HTTP OK: HTTP/1.1 200 OK - 75493 bytes in 3.092 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [06:55:48] PROBLEM - PHP7 rendering on mw1360 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200502T0700) [07:01:44] RECOVERY - MediaWiki memcached error rate on icinga1001 is OK: (C)5000 gt (W)1000 gt 951 https://wikitech.wikimedia.org/wiki/Memcached https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=1&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [07:03:56] PROBLEM - rsyslog in eqiad is failing to deliver messages on icinga1001 is CRITICAL: action=fwd_centrallog1001.eqiad.wmnet:6514 https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=eqiad+prometheus/ops [07:04:50] RECOVERY - PHP7 rendering on mw1360 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.375 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:07:55] !log asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 0 port 1 member 1 [07:08:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:08:30] !log asw2-d-eqiad> request virtual-chassis vc-port delete pic-slot 1 port 0 member 1 [07:08:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:08:36] RECOVERY - PHP7 rendering on mw1351 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.191 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:08:38] RECOVERY - PHP7 rendering on mw1357 is OK: HTTP OK: HTTP/1.1 200 OK - 75494 bytes in 5.647 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:08:42] RECOVERY - PHP7 rendering on mw1349 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.228 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:00] RECOVERY - PHP7 rendering on mw1350 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.193 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:10] RECOVERY - PHP7 rendering on mw1354 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.171 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:14] RECOVERY - PHP7 rendering on mw1361 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.172 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:14] RECOVERY - PHP7 rendering on mw1355 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.269 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:20] RECOVERY - PHP7 rendering on mw1359 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.163 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:20] RECOVERY - PHP7 rendering on mw1356 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.183 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:09:38] RECOVERY - Disk space on ores1003 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=ores1003&var-datasource=eqiad+prometheus/ops [07:09:44] RECOVERY - PHP7 rendering on mw1352 is OK: HTTP OK: HTTP/1.1 200 OK - 75492 bytes in 0.237 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [07:11:44] RECOVERY - MariaDB Slave Lag: s4 on db1125 is OK: OK slave_sql_lag Replication lag: 0.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave [07:12:02] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [07:12:10] RECOVERY - configured eth on ores1003 is OK: OK - interfaces up https://wikitech.wikimedia.org/wiki/Monitoring/check_eth [07:12:46] PROBLEM - Juniper virtual chassis ports on asw2-d-eqiad is CRITICAL: CRIT: Down: 1 Unknown: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23VCP_status [07:15:12] RECOVERY - rsyslog in codfw is failing to deliver messages on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=codfw+prometheus/ops [07:16:54] RECOVERY - rsyslog in eqiad is failing to deliver messages on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=eqiad+prometheus/ops [07:19:02] RECOVERY - Check whether ferm is active by checking the default input chain on ores1003 is OK: OK ferm input default policy is set https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [07:21:30] RECOVERY - Kafka Broker Under Replicated Partitions on kafka-jumbo1001 is OK: (C)10 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1001 [07:21:50] RECOVERY - Kafka Broker Under Replicated Partitions on kafka-jumbo1004 is OK: (C)10 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1004 [07:22:32] RECOVERY - Kafka Broker Under Replicated Partitions on kafka-jumbo1002 is OK: (C)10 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1002 [07:28:05] yesssss [07:29:49] (03CR) 10Elukey: [V: 03+2 C: 03+2] Refactor the exporter to support metrics specs via config file [software/druid_exporter] - 10https://gerrit.wikimedia.org/r/592261 (owner: 10Elukey) [07:38:34] 10Operations, 10netops: D1<->D8 link failure - https://phabricator.wikimedia.org/T251663 (10ayounsi) p:05Triage→03High [07:39:01] 10Operations, 10netops, 10serviceops: Investigate D1 appservers<->memcache TKOs - https://phabricator.wikimedia.org/T251601 (10ayounsi) [07:39:06] 10Operations, 10netops: D1<->D8 link failure - https://phabricator.wikimedia.org/T251663 (10ayounsi) [07:39:10] 10Operations, 10netops: D1<->D8 VC link failure - https://phabricator.wikimedia.org/T251663 (10ayounsi) [07:42:43] 10Operations, 10netops: D1<->D8 VC link failure - https://phabricator.wikimedia.org/T251663 (10ayounsi) [07:49:04] RECOVERY - Kafka Broker Replica Max Lag on kafka-jumbo1006 is OK: (C)5e+06 ge (W)1e+06 ge 2.342e+05 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/dashboard/db/kafka?panelId=16&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=jumbo-eqiad&var-kafka_broker=kafka-jumbo1006 [07:49:58] !log oblivian@cumin1001 conftool action : set/pooled=yes; selector: name=mw13(49|5[0-9]|6[0-2])\.eqiad\.wmnet [07:49:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:51:50] PROBLEM - mediawiki originals uploads -hourly- for eqiad on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe1005:9112 job=statsd_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Swift/How_To%23mediawiki_originals_uploads https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [07:52:08] PROBLEM - mediawiki originals uploads -hourly- for codfw on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe2005:9112 job=statsd_exporter site=codfw https://wikitech.wikimedia.org/wiki/Swift/How_To%23mediawiki_originals_uploads https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [08:37:32] (03PS1) 10Elukey: prometheus::druid_exporter: upgrade to 0.8 [puppet] - 10https://gerrit.wikimedia.org/r/593885 [08:38:57] (03PS2) 10Elukey: prometheus::druid_exporter: upgrade to 0.8 [puppet] - 10https://gerrit.wikimedia.org/r/593885 [08:50:14] (03PS3) 10Elukey: prometheus::druid_exporter: upgrade to 0.8 [puppet] - 10https://gerrit.wikimedia.org/r/593885 [09:19:10] RECOVERY - Backup freshness on backup1001 is OK: Fresh: 93 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [09:33:47] 10Operations, 10Toolforge, 10Traffic, 10HTTPS, and 3 others: Migrate tools.wmflabs.org to https only (and set HSTS) - https://phabricator.wikimedia.org/T102367 (10QEDK) [09:39:26] PROBLEM - SSH on db2082.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [10:40:11] RECOVERY - SSH on db2082.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [10:52:24] (03PS1) 10Ema: Makefile: fix 'atskafka' target, add 'test' [software/atskafka] - 10https://gerrit.wikimedia.org/r/593888 [10:52:26] (03PS1) 10Ema: json: avoid panic on malformed fields [software/atskafka] - 10https://gerrit.wikimedia.org/r/593889 [10:52:28] (03PS1) 10Ema: Add tests for atskafka.go [software/atskafka] - 10https://gerrit.wikimedia.org/r/593890 [11:05:26] (03PS1) 10Ema: Add basic prometeus integration [software/atskafka] - 10https://gerrit.wikimedia.org/r/593891 [11:05:28] (03PS1) 10Ema: Add pprof [software/atskafka] - 10https://gerrit.wikimedia.org/r/593892 [11:06:25] (03CR) 10jerkins-bot: [V: 04-1] Add basic prometeus integration [software/atskafka] - 10https://gerrit.wikimedia.org/r/593891 (owner: 10Ema) [11:06:27] (03CR) 10jerkins-bot: [V: 04-1] Add pprof [software/atskafka] - 10https://gerrit.wikimedia.org/r/593892 (owner: 10Ema) [11:10:02] (03PS1) 10Diomidis Spinellis: Correct typo in Greek Wikiversity logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593893 (https://phabricator.wikimedia.org/T248391) [11:10:03] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593893 (https://phabricator.wikimedia.org/T248391) (owner: 10Diomidis Spinellis) [11:11:23] (03PS2) 10Ema: Add basic prometeus integration [software/atskafka] - 10https://gerrit.wikimedia.org/r/593891 [11:11:25] (03PS2) 10Ema: Add pprof [software/atskafka] - 10https://gerrit.wikimedia.org/r/593892 [11:11:27] (03PS1) 10Ema: Add license and copyright notices [software/atskafka] - 10https://gerrit.wikimedia.org/r/593894 [11:12:46] (03CR) 10QEDK: [C: 03+1] "Recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593893 (https://phabricator.wikimedia.org/T248391) (owner: 10Diomidis Spinellis) [11:14:35] (03CR) 10QEDK: [C: 03+1] "LGTM; scheduling for SWAT May 4" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593893 (https://phabricator.wikimedia.org/T248391) (owner: 10Diomidis Spinellis) [11:18:44] (03CR) 10Diomidis Spinellis: "Thank your for prompting me to contribute the patch. I didn't know this change was handled through the repository. It seems logical in r" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593893 (https://phabricator.wikimedia.org/T248391) (owner: 10Diomidis Spinellis) [11:28:09] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3056 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [11:29:51] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3056 is OK: HTTP OK: HTTP/1.0 200 OK - 22733 bytes in 0.258 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [12:28:45] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3058 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [12:32:17] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3058 is OK: HTTP OK: HTTP/1.0 200 OK - 22721 bytes in 0.272 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [12:39:59] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=swagger_check_cxserver_cluster_eqiad site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [12:41:49] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [12:48:24] (03PS1) 10QEDK: Update Phab task for elwikiversity logos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593902 (https://phabricator.wikimedia.org/T248391) [12:48:55] RECOVERY - Maps - OSM synchronization lag - codfw on icinga1001 is OK: (C)2.592e+05 ge (W)1.764e+05 ge 9.293e+04 https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=12&fullscreen&orgId=1 [12:55:37] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3064 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [13:04:41] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3064 is OK: HTTP OK: HTTP/1.0 200 OK - 22731 bytes in 0.259 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [13:25:19] (03CR) 10Urbanecm: [C: 04-1] "this should add an entry to wmf-config/InterwikiSortOrders.php" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) (owner: 10Zoranzoki21) [13:27:12] (03CR) 10Reedy: [C: 04-1] "Generally it's easier to not add the row to wikiversons.json, as it causes a conflict if not merged pretty quickly, and they're just more " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) (owner: 10Zoranzoki21) [14:07:54] 10Operations, 10Security-Team, 10Patch-For-Review, 10User-jbond: Determine any impacts to SRE from OIT's planned move to JumpCloud for LDAP - https://phabricator.wikimedia.org/T244792 (10faidon) Happy to meet! Perhaps it would also help if I could provide some broader context in writing as food for thought... [14:26:11] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3056 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [14:37:05] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3056 is OK: HTTP OK: HTTP/1.0 200 OK - 22729 bytes in 0.257 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [15:03:50] (03CR) 10Cmjohnson: [C: 03+2] Adding netboot.cfg and dhcpd for cloudelastic100[56] [puppet] - 10https://gerrit.wikimedia.org/r/593613 (https://phabricator.wikimedia.org/T249062) (owner: 10Cmjohnson) [15:05:12] (03PS3) 10Cmjohnson: Adding netboot.cfg and dhcpd for cloudelastic100[56] [puppet] - 10https://gerrit.wikimedia.org/r/593613 (https://phabricator.wikimedia.org/T249062) [15:05:37] (03CR) 10Cmjohnson: [V: 03+2 C: 03+2] Adding netboot.cfg and dhcpd for cloudelastic100[56] [puppet] - 10https://gerrit.wikimedia.org/r/593613 (https://phabricator.wikimedia.org/T249062) (owner: 10Cmjohnson) [15:24:45] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=pdu_sentry4 site=eqsin https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [15:26:35] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [15:32:07] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=swagger_check_cxserver_cluster_eqiad site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [15:33:57] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [15:43:30] 10Operations, 10Research, 10Patch-For-Review: recommendation api's test on scb nodes are flapping - https://phabricator.wikimedia.org/T247732 (10bmansurov) @elukey Please take a look at the above patch. Rather than stubbing requests to MW API (and thus making a lot of code changes) I decided not to log healt... [15:46:19] 10Operations, 10Research, 10Patch-For-Review: recommendation api's test on scb nodes are flapping - https://phabricator.wikimedia.org/T247732 (10elukey) >>! In T247732#6102300, @bmansurov wrote: > @elukey Please take a look at the above patch. Rather than stubbing requests to MW API (and thus making a lot of... [15:53:38] 10Operations, 10Research, 10Patch-For-Review: recommendation api's test on scb nodes are flapping - https://phabricator.wikimedia.org/T247732 (10bmansurov) OK, I see. Then I'll not merge that patch. I'll handle the response code directly, possibly returning a 404 for failed health check requests. [16:04:15] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3064 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [16:13:15] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3064 is OK: HTTP OK: HTTP/1.0 200 OK - 22732 bytes in 0.257 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [17:13:39] (03CR) 10Zoranzoki21: "I no see edit button in new interface, where is it?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) (owner: 10Zoranzoki21) [17:14:19] (03CR) 10Zoranzoki21: "> Patch Set 2: Code-Review-1" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) (owner: 10Zoranzoki21) [17:14:32] (03PS3) 10Zoranzoki21: Initial config for awawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) [17:15:20] (03CR) 10Zoranzoki21: Initial config for awawiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) (owner: 10Zoranzoki21) [17:16:20] (03CR) 10Zoranzoki21: "> Patch Set 2: Code-Review-1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593743 (https://phabricator.wikimedia.org/T251371) (owner: 10Zoranzoki21) [17:36:28] (03PS1) 10AntiCompositeNumber: engine.vips: fall back to tinyrgb when libpng complains [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/593920 (https://phabricator.wikimedia.org/T219569) [18:41:15] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3054 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [18:43:56] ACKNOWLEDGEMENT - Juniper virtual chassis ports on asw2-d-eqiad is CRITICAL: CRIT: Down: 1 Unknown: 0 Ayounsi https://phabricator.wikimedia.org/T251663 https://wikitech.wikimedia.org/wiki/Network_monitoring%23VCP_status [18:52:07] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3054 is OK: HTTP OK: HTTP/1.0 200 OK - 22721 bytes in 0.287 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [20:33:43] (03CR) 10QEDK: [C: 03+1] "> Patch Set 1:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593893 (https://phabricator.wikimedia.org/T248391) (owner: 10Diomidis Spinellis) [21:36:37] (03PS7) 10Urbanecm: noc.wikimedia.org: highlight.php should not append .txt to dblist URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/591459 (https://phabricator.wikimedia.org/T250852) [21:36:45] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=swagger_check_restbase_esams site=esams https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [21:37:32] (03CR) 10jerkins-bot: [V: 04-1] noc.wikimedia.org: highlight.php should not append .txt to dblist URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/591459 (https://phabricator.wikimedia.org/T250852) (owner: 10Urbanecm) [21:38:37] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [21:39:38] (03PS8) 10Urbanecm: noc.wikimedia.org: highlight.php should not append .txt to dblist URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/591459 (https://phabricator.wikimedia.org/T250852) [21:40:08] (03CR) 10Urbanecm: noc.wikimedia.org: highlight.php should not append .txt to dblist URLs (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/591459 (https://phabricator.wikimedia.org/T250852) (owner: 10Urbanecm) [21:40:44] (03CR) 10jerkins-bot: [V: 04-1] noc.wikimedia.org: highlight.php should not append .txt to dblist URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/591459 (https://phabricator.wikimedia.org/T250852) (owner: 10Urbanecm) [21:43:50] (03PS9) 10Urbanecm: noc.wikimedia.org: highlight.php should not append .txt to dblist URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/591459 (https://phabricator.wikimedia.org/T250852) [21:47:52] (03PS1) 10Urbanecm: noc: Remove broken symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/593929 [22:12:05] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3050 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [22:17:27] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3050 is OK: HTTP OK: HTTP/1.0 200 OK - 22727 bytes in 0.258 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server