[00:07:37] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [00:25:27] RECOVERY - Persistent high iowait on labstore1001 is OK Less than 50.00% above the threshold [25.0] [00:40:40] (03CR) 10Krinkle: [C: 031] Removed "refreshLinks" from $wgJobBackoffThrottling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/210857 (owner: 10Aaron Schulz) [01:15:47] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [01:35:17] PROBLEM - are wikitech and wt-static in sync on silver is CRITICAL: wikitech-static CRIT - wikitech and wikitech-static out of sync (91196s 90000s) [01:38:37] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [01:56:17] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.14% of data above the critical threshold [500.0] [01:56:27] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [01:58:26] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL Anomaly detected: 11 data above and 3 below the confidence bounds [02:17:27] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [02:21:48] !log l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 06m 24s) [02:22:04] Logged the message, Master [02:26:58] !log LocalisationUpdate completed (1.26wmf5) at 2015-05-18 02:25:54+00:00 [02:27:03] Logged the message, Master [02:42:21] !log l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 05m 35s) [02:42:30] Logged the message, Master [02:46:55] !log LocalisationUpdate completed (1.26wmf6) at 2015-05-18 02:45:52+00:00 [02:47:00] Logged the message, Master [03:35:38] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [03:37:46] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK No anomaly detected [03:51:53] 6operations, 7database: investigate performance_schema for wmf prod - https://phabricator.wikimedia.org/T99485#1292599 (10Springle) 3NEW a:3jcrespo [03:59:48] 6operations, 7database: switch to innodb tables for replication state - https://phabricator.wikimedia.org/T99486#1292611 (10Springle) 3NEW a:3jcrespo [04:05:24] 6operations, 6Phabricator: m3 set max_allowed_packet to 33554432 or greater - https://phabricator.wikimedia.org/T98339#1292618 (10Springle) a:5Springle>3jcrespo [04:11:37] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [04:20:23] 6operations, 6Phabricator: m3 set max_allowed_packet to 33554432 or greater - https://phabricator.wikimedia.org/T98339#1292629 (10Springle) If we change this, check mysqldump command for m3 backups on dbstore1001. [04:21:27] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [04:25:16] 6operations, 7HHVM, 5Patch-For-Review: investigate HHVM mysqlExtension::ConnectTimeout - https://phabricator.wikimedia.org/T98489#1292630 (10Springle) p:5Normal>3High [04:25:38] PROBLEM - puppet last run on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:25:46] PROBLEM - RAID on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:27:17] RECOVERY - RAID on mw1107 is OK no RAID installed [04:28:56] RECOVERY - puppet last run on mw1107 is OK Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:36:06] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0] [04:41:07] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0] [05:03:57] RECOVERY - are wikitech and wt-static in sync on silver is OK: wikitech-static OK - wikitech and wikitech-static in sync (12590 90000s) [05:18:54] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon May 18 05:17:50 UTC 2015 (duration 17m 49s) [05:19:00] Logged the message, Master [05:20:08] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0] [05:23:28] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [05:39:07] PROBLEM - Host eeden is DOWN: PING CRITICAL - Packet loss = 100% [05:39:47] PROBLEM - Host ns2-v4 is DOWN: PING CRITICAL - Packet loss = 100% [05:50:38] PROBLEM - Varnishkafka Delivery Errors per minute on cp4015 is CRITICAL 11.11% of data above the critical threshold [20000.0] [05:53:49] hmm, cologneblue is missing on mw.o [05:55:36] RECOVERY - Varnishkafka Delivery Errors per minute on cp4015 is OK Less than 1.00% above the threshold [0.0] [06:00:36] RECOVERY - Host eeden is UPING OK - Packet loss = 0%, RTA = 88.63 ms [06:00:47] RECOVERY - Host ns2-v4 is UPING OK - Packet loss = 0%, RTA = 89.21 ms [06:01:15] (03PS1) 10Florianschmidtwelzow: Remove gather-hidelist from AvailableRights [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211663 (https://phabricator.wikimedia.org/T94652) [06:03:25] (03CR) 10Legoktm: "Is the extension deployed on meta?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211663 (https://phabricator.wikimedia.org/T94652) (owner: 10Florianschmidtwelzow) [06:16:49] (03CR) 10Jalexander: [C: 04-1] "Please do not merge, this extension is not installed on meta and without it being specifically added to availablerights it is not able to " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211663 (https://phabricator.wikimedia.org/T94652) (owner: 10Florianschmidtwelzow) [06:30:27] PROBLEM - puppet last run on logstash1006 is CRITICAL puppet fail [06:31:17] PROBLEM - puppet last run on mw1100 is CRITICAL Puppet has 1 failures [06:31:17] PROBLEM - puppet last run on analytics1030 is CRITICAL Puppet has 1 failures [06:31:48] PROBLEM - puppet last run on cp4008 is CRITICAL Puppet has 2 failures [06:31:48] PROBLEM - puppet last run on cp4004 is CRITICAL Puppet has 1 failures [06:32:17] PROBLEM - puppet last run on mw1003 is CRITICAL Puppet has 1 failures [06:32:47] PROBLEM - puppet last run on mw1042 is CRITICAL Puppet has 1 failures [06:32:48] PROBLEM - puppet last run on lvs2004 is CRITICAL Puppet has 1 failures [06:33:26] PROBLEM - puppet last run on db2065 is CRITICAL Puppet has 1 failures [06:33:56] PROBLEM - puppet last run on mw1065 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw2206 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw2143 is CRITICAL Puppet has 1 failures [06:35:37] PROBLEM - puppet last run on mw2113 is CRITICAL Puppet has 1 failures [06:35:53] (03PS5) 10Giuseppe Lavagetto: hiera: use the proxy backend, rationalize the hierarchy [puppet] - 10https://gerrit.wikimedia.org/r/207129 [06:38:27] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [06:45:27] RECOVERY - puppet last run on mw1003 is OK Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:45:57] RECOVERY - puppet last run on lvs2004 is OK Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on mw1100 is OK Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:46:07] RECOVERY - puppet last run on analytics1030 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:46:37] RECOVERY - puppet last run on db2065 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:46:37] RECOVERY - puppet last run on cp4008 is OK Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on logstash1006 is OK Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:47:06] RECOVERY - puppet last run on mw1065 is OK Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw2206 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw2143 is OK Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw2113 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on mw1042 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:07] (03CR) 10Giuseppe Lavagetto: [C: 032] "I tested this with the puppet compiler and it results to be a no-op - the only changes are related to ganglia_class apparently." [puppet] - 10https://gerrit.wikimedia.org/r/207129 (owner: 10Giuseppe Lavagetto) [06:48:17] RECOVERY - puppet last run on cp4004 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:50:04] 6operations, 10Hackathon-Lyon-2015, 10Wikimedia-Site-requests, 7I18n, 7Tracking: Wikis waiting to be renamed (tracking) - https://phabricator.wikimedia.org/T21986#1292743 (10jcrespo) [06:51:37] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0] [06:54:37] PROBLEM - puppet last run on mw1229 is CRITICAL puppet fail [06:54:52] (03PS1) 10Giuseppe Lavagetto: Revert "hiera: use the proxy backend, rationalize the hierarchy" [puppet] - 10https://gerrit.wikimedia.org/r/211666 [06:55:01] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Revert "hiera: use the proxy backend, rationalize the hierarchy" [puppet] - 10https://gerrit.wikimedia.org/r/211666 (owner: 10Giuseppe Lavagetto) [06:55:08] PROBLEM - puppet last run on mw1032 is CRITICAL puppet fail [06:55:09] <_joe_> fu.ck [06:55:17] PROBLEM - puppet last run on mw1185 is CRITICAL puppet fail [06:55:26] PROBLEM - puppet last run on mw2153 is CRITICAL puppet fail [06:55:27] PROBLEM - puppet last run on mw2209 is CRITICAL puppet fail [06:55:27] PROBLEM - puppet last run on mw2175 is CRITICAL puppet fail [06:55:27] PROBLEM - puppet last run on mw2140 is CRITICAL puppet fail [06:55:27] PROBLEM - puppet last run on mw2204 is CRITICAL puppet fail [06:55:27] PROBLEM - puppet last run on mw2144 is CRITICAL puppet fail [06:55:27] PROBLEM - puppet last run on mw2201 is CRITICAL puppet fail [06:55:28] PROBLEM - puppet last run on mw2205 is CRITICAL puppet fail [06:55:28] PROBLEM - puppet last run on mw2009 is CRITICAL puppet fail [06:55:29] PROBLEM - puppet last run on elastic1023 is CRITICAL puppet fail [06:55:29] PROBLEM - puppet last run on ganeti2006 is CRITICAL puppet fail [06:55:36] PROBLEM - puppet last run on elastic1017 is CRITICAL puppet fail [06:55:37] PROBLEM - puppet last run on logstash1005 is CRITICAL puppet fail [06:55:38] <_joe_> that's me :( [06:55:47] PROBLEM - puppet last run on mw1043 is CRITICAL puppet fail [06:55:47] PROBLEM - puppet last run on mw1105 is CRITICAL puppet fail [06:55:47] PROBLEM - puppet last run on mw1033 is CRITICAL puppet fail [06:55:56] PROBLEM - puppet last run on mw2091 is CRITICAL puppet fail [06:55:57] PROBLEM - puppet last run on mw1225 is CRITICAL puppet fail [06:56:17] PROBLEM - puppet last run on mw1122 is CRITICAL puppet fail [06:56:17] PROBLEM - puppet last run on mw1024 is CRITICAL puppet fail [06:56:28] PROBLEM - puppet last run on iodine is CRITICAL puppet fail [06:56:36] PROBLEM - puppet last run on mw1077 is CRITICAL puppet fail [06:56:36] PROBLEM - puppet last run on mw1209 is CRITICAL puppet fail [06:56:48] <_joe_> I can't get why [06:56:57] PROBLEM - puppet last run on mw2188 is CRITICAL puppet fail [06:56:57] PROBLEM - puppet last run on mw2185 is CRITICAL puppet fail [06:56:57] PROBLEM - puppet last run on mw2180 is CRITICAL puppet fail [06:56:57] PROBLEM - puppet last run on mw2148 is CRITICAL puppet fail [06:57:05] <_joe_> Alex's compiler was telling me everything was ok, wtf [06:57:06] PROBLEM - puppet last run on mw2156 is CRITICAL puppet fail [06:57:07] PROBLEM - puppet last run on mw2157 is CRITICAL puppet fail [06:57:07] PROBLEM - puppet last run on mw2141 is CRITICAL puppet fail [06:57:07] PROBLEM - puppet last run on mw2069 is CRITICAL puppet fail [06:57:08] PROBLEM - puppet last run on mw2010 is CRITICAL puppet fail [06:57:08] PROBLEM - puppet last run on mw2063 is CRITICAL puppet fail [06:57:08] PROBLEM - puppet last run on ganeti2002 is CRITICAL puppet fail [06:57:08] PROBLEM - puppet last run on ganeti2004 is CRITICAL puppet fail [06:57:09] PROBLEM - puppet last run on ganeti2003 is CRITICAL puppet fail [06:57:09] PROBLEM - puppet last run on mw1142 is CRITICAL puppet fail [06:57:10] PROBLEM - puppet last run on mw1231 is CRITICAL puppet fail [06:57:36] PROBLEM - puppet last run on mw2075 is CRITICAL puppet fail [06:57:46] PROBLEM - puppet last run on mw1071 is CRITICAL puppet fail [06:57:57] PROBLEM - puppet last run on mw1215 is CRITICAL puppet fail [06:58:16] PROBLEM - puppet last run on mw1193 is CRITICAL puppet fail [06:58:16] PROBLEM - puppet last run on mw1253 is CRITICAL puppet fail [06:58:17] PROBLEM - puppet last run on elastic1003 is CRITICAL puppet fail [06:58:17] PROBLEM - puppet last run on mw1021 is CRITICAL puppet fail [06:58:18] PROBLEM - puppet last run on mw1086 is CRITICAL puppet fail [06:58:28] PROBLEM - puppet last run on mw1203 is CRITICAL puppet fail [06:58:37] PROBLEM - puppet last run on mw1090 is CRITICAL puppet fail [06:58:37] PROBLEM - puppet last run on mw1241 is CRITICAL puppet fail [06:58:37] PROBLEM - puppet last run on mw2158 is CRITICAL puppet fail [06:58:37] PROBLEM - puppet last run on mw2195 is CRITICAL puppet fail [06:58:38] PROBLEM - puppet last run on mw2137 is CRITICAL puppet fail [06:58:46] PROBLEM - puppet last run on mw2129 is CRITICAL puppet fail [06:58:46] PROBLEM - puppet last run on mw2128 is CRITICAL puppet fail [06:58:46] PROBLEM - puppet last run on mw2078 is CRITICAL puppet fail [06:58:46] PROBLEM - puppet last run on mw1030 is CRITICAL puppet fail [06:58:46] PROBLEM - puppet last run on mw2060 is CRITICAL puppet fail [06:58:47] PROBLEM - puppet last run on mw2021 is CRITICAL puppet fail [06:58:47] PROBLEM - puppet last run on mw2019 is CRITICAL puppet fail [06:58:47] PROBLEM - puppet last run on mw2067 is CRITICAL puppet fail [06:58:48] PROBLEM - puppet last run on mw2040 is CRITICAL puppet fail [06:58:49] PROBLEM - puppet last run on mw1037 is CRITICAL puppet fail [06:58:49] PROBLEM - puppet last run on mw1135 is CRITICAL puppet fail [06:58:56] PROBLEM - puppet last run on mw1220 is CRITICAL puppet fail [06:58:58] PROBLEM - puppet last run on mw1104 is CRITICAL puppet fail [06:59:07] PROBLEM - puppet last run on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:17] PROBLEM - puppet last run on mw2018 is CRITICAL puppet fail [06:59:17] PROBLEM - puppet last run on mw2039 is CRITICAL puppet fail [06:59:36] PROBLEM - puppet last run on restbase1003 is CRITICAL puppet fail [06:59:36] PROBLEM - puppet last run on mw1154 is CRITICAL puppet fail [06:59:46] PROBLEM - puppet last run on cp3007 is CRITICAL puppet fail [06:59:46] PROBLEM - puppet last run on cp3038 is CRITICAL puppet fail [06:59:46] PROBLEM - puppet last run on cp3017 is CRITICAL puppet fail [06:59:57] PROBLEM - puppet last run on mw1158 is CRITICAL puppet fail [07:00:07] PROBLEM - puppet last run on mw1047 is CRITICAL puppet fail [07:00:07] PROBLEM - puppet last run on mw1018 is CRITICAL puppet fail [07:00:17] PROBLEM - puppet last run on lanthanum is CRITICAL puppet fail [07:00:17] PROBLEM - puppet last run on db2055 is CRITICAL Puppet has 8 failures [07:00:17] PROBLEM - puppet last run on mw2211 is CRITICAL puppet fail [07:00:17] PROBLEM - puppet last run on mw2142 is CRITICAL puppet fail [07:00:17] PROBLEM - puppet last run on mw1128 is CRITICAL puppet fail [07:00:18] PROBLEM - puppet last run on mw2131 is CRITICAL puppet fail [07:00:18] PROBLEM - puppet last run on mw2101 is CRITICAL puppet fail [07:00:19] PROBLEM - puppet last run on mw1073 is CRITICAL puppet fail [07:00:19] PROBLEM - puppet last run on mw2110 is CRITICAL puppet fail [07:00:26] PROBLEM - puppet last run on mw2070 is CRITICAL Puppet has 8 failures [07:00:26] PROBLEM - puppet last run on mw2055 is CRITICAL puppet fail [07:00:26] PROBLEM - puppet last run on db2017 is CRITICAL Puppet has 6 failures [07:00:27] PROBLEM - puppet last run on mw2077 is CRITICAL puppet fail [07:00:27] PROBLEM - puppet last run on elastic1025 is CRITICAL puppet fail [07:00:56] PROBLEM - puppet last run on lvs3002 is CRITICAL puppet fail [07:01:18] PROBLEM - puppet last run on cp3048 is CRITICAL Puppet has 4 failures [07:02:46] (03Abandoned) 10Florianschmidtwelzow: Remove gather-hidelist from AvailableRights [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211663 (https://phabricator.wikimedia.org/T94652) (owner: 10Florianschmidtwelzow) [07:02:58] PROBLEM - puppet last run on cp4010 is CRITICAL Puppet has 3 failures [07:04:47] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0] [07:05:17] PROBLEM - puppet last run on wtp2016 is CRITICAL Puppet has 6 failures [07:05:26] PROBLEM - puppet last run on eventlog2001 is CRITICAL Puppet has 5 failures [07:06:56] PROBLEM - puppet last run on db2047 is CRITICAL Puppet has 1 failures [07:06:57] PROBLEM - puppet last run on mw2176 is CRITICAL Puppet has 8 failures [07:06:57] PROBLEM - puppet last run on db2010 is CRITICAL Puppet has 5 failures [07:10:47] PROBLEM - RAID on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:41] RECOVERY - puppet last run on logstash1005 is OK Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:13:11] PROBLEM - puppet last run on mw2084 is CRITICAL Puppet has 7 failures [07:13:21] RECOVERY - puppet last run on mw1030 is OK Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:14:01] PROBLEM - puppet last run on mw2024 is CRITICAL Puppet has 2 failures [07:14:01] PROBLEM - puppet last run on mw2047 is CRITICAL Puppet has 4 failures [07:14:02] RECOVERY - puppet last run on ganeti2004 is OK Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:14:02] RECOVERY - puppet last run on ganeti2006 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:10] RECOVERY - puppet last run on ganeti2002 is OK Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:14:10] RECOVERY - puppet last run on ganeti2003 is OK Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:14:10] RECOVERY - puppet last run on elastic1023 is OK Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:14:11] RECOVERY - puppet last run on elastic1017 is OK Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:14:31] RECOVERY - puppet last run on elastic1003 is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:14:40] RECOVERY - puppet last run on mw1229 is OK Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:14:40] RECOVERY - puppet last run on mw1043 is OK Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:14:41] RECOVERY - puppet last run on mw1033 is OK Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:14:41] RECOVERY - puppet last run on mw1105 is OK Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:14:41] RECOVERY - puppet last run on mw2201 is OK Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:14:50] RECOVERY - puppet last run on mw2091 is OK Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:14:51] RECOVERY - puppet last run on mw1032 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:14:51] RECOVERY - puppet last run on mw1215 is OK Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:14:51] PROBLEM - puppet last run on mw2052 is CRITICAL Puppet has 1 failures [07:14:51] PROBLEM - puppet last run on mw2120 is CRITICAL Puppet has 4 failures [07:14:51] RECOVERY - puppet last run on mw2148 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:15:01] RECOVERY - puppet last run on mw1225 is OK Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:15:01] RECOVERY - puppet last run on mw1071 is OK Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:15:02] RECOVERY - puppet last run on iodine is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:11] RECOVERY - puppet last run on mw1122 is OK Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:15:11] RECOVERY - puppet last run on mw2180 is OK Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:15:11] RECOVERY - puppet last run on mw2144 is OK Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:15:11] RECOVERY - puppet last run on mw2205 is OK Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:15:11] RECOVERY - puppet last run on mw2209 is OK Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:15:21] RECOVERY - puppet last run on mw1086 is OK Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:15:21] RECOVERY - puppet last run on mw2157 is OK Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:15:30] RECOVERY - puppet last run on mw1185 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:30] RECOVERY - puppet last run on mw1209 is OK Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:15:30] RECOVERY - puppet last run on mw2156 is OK Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:15:30] PROBLEM - puppet last run on mw2182 is CRITICAL Puppet has 1 failures [07:15:31] RECOVERY - puppet last run on mw1024 is OK Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:15:31] RECOVERY - puppet last run on mw2175 is OK Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:15:32] RECOVERY - puppet last run on mw2140 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:41] RECOVERY - puppet last run on mw2069 is OK Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:15:41] RECOVERY - puppet last run on mw2010 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:15:42] RECOVERY - puppet last run on mw1037 is OK Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:15:42] RECOVERY - puppet last run on mw2040 is OK Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:15:42] RECOVERY - puppet last run on mw2063 is OK Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:15:42] RECOVERY - puppet last run on mw1135 is OK Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:15:42] RECOVERY - puppet last run on mw1142 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:43] RECOVERY - puppet last run on mw1231 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:43] RECOVERY - puppet last run on mw1220 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:50] RECOVERY - puppet last run on mw1077 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:50] RECOVERY - puppet last run on mw2204 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:16:00] RECOVERY - puppet last run on mw1241 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:00] RECOVERY - puppet last run on mw1090 is OK Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:16:01] RECOVERY - puppet last run on mw2128 is OK Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:16:01] RECOVERY - puppet last run on mw2153 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:01] RECOVERY - puppet last run on mw1104 is OK Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:16:10] RECOVERY - puppet last run on cp3007 is OK Puppet is currently enabled, last run 1 second ago with 0 failures [07:16:11] RECOVERY - puppet last run on mw1154 is OK Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:16:11] RECOVERY - puppet last run on mw1203 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:12] RECOVERY - puppet last run on mw2009 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:30] RECOVERY - puppet last run on mw1253 is OK Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:16:30] RECOVERY - puppet last run on mw2075 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:30] RECOVERY - puppet last run on mw1021 is OK Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:16:31] RECOVERY - puppet last run on mw2188 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:31] RECOVERY - puppet last run on db2047 is OK Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:16:31] RECOVERY - puppet last run on mw2141 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:31] RECOVERY - puppet last run on mw2185 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:40] RECOVERY - puppet last run on mw1158 is OK Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:16:40] PROBLEM - puppet last run on mw1199 is CRITICAL Puppet has 5 failures [07:16:41] RECOVERY - puppet last run on db2055 is OK Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:16:50] RECOVERY - puppet last run on db2010 is OK Puppet is currently enabled, last run 1 second ago with 0 failures [07:16:50] PROBLEM - puppet last run on cp4020 is CRITICAL puppet fail [07:16:50] RECOVERY - puppet last run on mw1193 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:01] RECOVERY - puppet last run on mw2176 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:10] RECOVERY - puppet last run on restbase1003 is OK Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:17:22] RECOVERY - puppet last run on mw2060 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:22] RECOVERY - puppet last run on db2017 is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:17:22] RECOVERY - puppet last run on mw2021 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:22] RECOVERY - puppet last run on mw2055 is OK Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:17:22] RECOVERY - puppet last run on mw2070 is OK Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:17:22] RECOVERY - puppet last run on mw2019 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:23] RECOVERY - puppet last run on eventlog2001 is OK Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:17:24] RECOVERY - puppet last run on mw2067 is OK Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:17:24] RECOVERY - puppet last run on elastic1025 is OK Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:17:30] RECOVERY - puppet last run on mw1073 is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:17:30] RECOVERY - puppet last run on cp3048 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:17:30] RECOVERY - puppet last run on mw2137 is OK Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:17:40] RECOVERY - puppet last run on mw1047 is OK Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:17:41] RECOVERY - puppet last run on mw1128 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:17:41] RECOVERY - puppet last run on mw2078 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:51] RECOVERY - puppet last run on mw2158 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:51] RECOVERY - puppet last run on mw2195 is OK Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:18:01] RECOVERY - puppet last run on cp4010 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:02] RECOVERY - puppet last run on cp3017 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:02] RECOVERY - puppet last run on mw2018 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:02] RECOVERY - puppet last run on mw2039 is OK Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:18:02] RECOVERY - puppet last run on mw2084 is OK Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:18:02] RECOVERY - puppet last run on mw2129 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:04] (03CR) 10Mobrovac: "Ok, the deploy repo has been updated to the latest mathoid code in Iea3c5e6002123b38ad52214b26189d59569ac287 so this should be good to go " [puppet] - 10https://gerrit.wikimedia.org/r/167413 (https://phabricator.wikimedia.org/T97124) (owner: 10Ori.livneh) [07:18:10] RECOVERY - puppet last run on mw2142 is OK Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:18:11] RECOVERY - puppet last run on mw2052 is OK Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:18:11] RECOVERY - puppet last run on mw2120 is OK Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:18:11] RECOVERY - puppet last run on mw2131 is OK Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:18:12] RECOVERY - puppet last run on mw1199 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:31] RECOVERY - puppet last run on wtp2016 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:40] RECOVERY - puppet last run on lanthanum is OK Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:18:41] RECOVERY - puppet last run on mw1018 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:50] RECOVERY - puppet last run on mw2182 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:01] RECOVERY - puppet last run on cp3038 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:01] RECOVERY - puppet last run on mw2211 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:01] RECOVERY - puppet last run on mw2024 is OK Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:19:02] RECOVERY - puppet last run on mw2047 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:02] RECOVERY - puppet last run on mw2077 is OK Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:19:10] RECOVERY - puppet last run on mw2110 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:21] RECOVERY - puppet last run on mw2101 is OK Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:20:01] RECOVERY - puppet last run on lvs3002 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:48] RECOVERY - RAID on mw1107 is OK no RAID installed [07:21:59] PROBLEM - puppet last run on mw2103 is CRITICAL puppet fail [07:21:59] RECOVERY - puppet last run on mw1107 is OK Puppet is currently enabled, last run 5 minutes ago with 0 failures [07:25:25] (03PS1) 10Jcrespo: Set max_allowed_packet on m3 (phabricator) to 32M (current: 16M) [puppet] - 10https://gerrit.wikimedia.org/r/211669 (https://phabricator.wikimedia.org/T98339) [07:30:21] (03CR) 10Santhosh: [C: 031] CX: Enable 'cxstats' campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211116 (owner: 10KartikMistry) [07:34:19] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0] [07:34:19] RECOVERY - puppet last run on cp4020 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:49] RECOVERY - puppet last run on mw2103 is OK Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:43:18] (03PS2) 10Jcrespo: Set max_allowed_packet on m3 (phabricator) to 32M (current: 16M) [puppet] - 10https://gerrit.wikimedia.org/r/211669 (https://phabricator.wikimedia.org/T98339) [07:51:11] (03PS1) 10KartikMistry: CX: Add wikis for deployment on 20150518 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211671 (https://phabricator.wikimedia.org/T98454) [07:51:17] (03CR) 10jenkins-bot: [V: 04-1] CX: Add wikis for deployment on 20150518 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211671 (https://phabricator.wikimedia.org/T98454) (owner: 10KartikMistry) [07:53:35] (03PS2) 10KartikMistry: CX: Add wikis for deployment on 20150518 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/211671 (https://phabricator.wikimedia.org/T98454) [07:54:40] (03CR) 10Springle: [C: 031] Set max_allowed_packet on m3 (phabricator) to 32M (current: 16M) [puppet] - 10https://gerrit.wikimedia.org/r/211669 (https://phabricator.wikimedia.org/T98339) (owner: 10Jcrespo) [07:54:46] 6operations, 6Phabricator, 5Patch-For-Review: m3 set max_allowed_packet to 33554432 or greater - https://phabricator.wikimedia.org/T98339#1292795 (10jcrespo) Once the patch is applied, the second part of the fix will be to execute `SET GLOBAL max_allowed_packet = 32M;` from slaves to masters on db2012.codfw.... [08:01:39] PROBLEM - puppet last run on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:04:07] (03CR) 10Jcrespo: [C: 032] Set max_allowed_packet on m3 (phabricator) to 32M (current: 16M) [puppet] - 10https://gerrit.wikimedia.org/r/211669 (https://phabricator.wikimedia.org/T98339) (owner: 10Jcrespo) [08:04:19] PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0] [08:05:50] (03PS1) 10Filippo Giunchedi: es-tool: wait longer before enabling replication [puppet] - 10https://gerrit.wikimedia.org/r/211672 (https://phabricator.wikimedia.org/T99500) [08:06:38] PROBLEM - RAID on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:13:10] RECOVERY - RAID on mw1107 is OK no RAID installed [08:13:10] RECOVERY - puppet last run on mw1107 is OK Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:20:46] (03CR) 10Giuseppe Lavagetto: [C: 031] Set HHVM mysql connection timeout to 3s [puppet] - 10https://gerrit.wikimedia.org/r/211155 (https://phabricator.wikimedia.org/T98489) (owner: 10BryanDavis) [08:22:53] mobrovac: bonjour :} [08:23:08] mobrovac: are you handling the beta cluster VE/RestBase path config issue? https://phabricator.wikimedia.org/T99496 [08:23:10] buongiorno mr hashar [08:24:01] (03CR) 10Giuseppe Lavagetto: "I'm not sure how much of a burden on the puppetmasters would be having this running for every puppet run would cause. I guess quite a lot." [puppet] - 10https://gerrit.wikimedia.org/r/210926 (owner: 10Faidon Liambotis) [08:24:08] hashar: for now i'm looking into why RB keeps failing there, the path config should be easy enough to fix after that [08:24:13] (famous last words...) [08:27:56] ahah [08:28:03] mobrovac: mind if I assign the task to you ? [08:28:12] oh you did [08:28:14] awesome [08:28:15] hashar: self-assigned already [08:28:16] :P [08:33:18] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.69% of data above the critical threshold [500.0] [08:38:59] PROBLEM - Varnishkafka Delivery Errors per minute on cp4014 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:39:28] PROBLEM - Varnishkafka Delivery Errors per minute on cp4007 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:39:58] 6operations, 10Wikimedia-DNS, 10Wikimedia-Video: Please set up a CNAME for videoserver.wikimedia.org to Video Editing Server - https://phabricator.wikimedia.org/T99216#1292858 (10Bawolff) >>! In T99216#1290616, @80686 wrote: > good point about the privacy policy. My suggestion is that we point at the WMF pri... [08:43:58] PROBLEM - BGP status on cr2-ulsfo is CRITICAL No response from remote host 198.35.26.193 [08:44:39] PROBLEM - Varnishkafka Delivery Errors per minute on cp4015 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:44:58] PROBLEM - Varnishkafka Delivery Errors per minute on cp4010 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:45:19] PROBLEM - Varnishkafka Delivery Errors per minute on cp4006 is CRITICAL 22.22% of data above the critical threshold [20000.0] [08:45:29] PROBLEM - Varnishkafka Delivery Errors per minute on cp4005 is CRITICAL 22.22% of data above the critical threshold [20000.0] [08:45:39] PROBLEM - Varnishkafka Delivery Errors per minute on cp4004 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:45:48] PROBLEM - Varnishkafka Delivery Errors per minute on cp4003 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:46:09] PROBLEM - Varnishkafka Delivery Errors per minute on cp4013 is CRITICAL 22.22% of data above the critical threshold [20000.0] [08:46:38] PROBLEM - Varnishkafka Delivery Errors per minute on cp4002 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:47:28] PROBLEM - Varnishkafka Delivery Errors per minute on cp4001 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:47:40] PROBLEM - Varnishkafka Delivery Errors per minute on cp4017 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:48:18] RECOVERY - Varnishkafka Delivery Errors per minute on cp4010 is OK Less than 1.00% above the threshold [0.0] [08:48:59] RECOVERY - BGP status on cr2-ulsfo is OK host 198.35.26.193, sessions up: 45, down: 0, shutdown: 0 [08:49:19] RECOVERY - Varnishkafka Delivery Errors per minute on cp4007 is OK Less than 1.00% above the threshold [0.0] [08:49:38] RECOVERY - Varnishkafka Delivery Errors per minute on cp4015 is OK Less than 1.00% above the threshold [0.0] [08:50:05] mmhh ulsfo problems? [08:50:39] RECOVERY - Varnishkafka Delivery Errors per minute on cp4004 is OK Less than 1.00% above the threshold [0.0] [08:50:39] RECOVERY - Varnishkafka Delivery Errors per minute on cp4014 is OK Less than 1.00% above the threshold [0.0] [08:50:40] RECOVERY - Varnishkafka Delivery Errors per minute on cp4003 is OK Less than 1.00% above the threshold [0.0] [08:50:48] RECOVERY - Varnishkafka Delivery Errors per minute on cp4001 is OK Less than 1.00% above the threshold [0.0] [08:50:59] RECOVERY - Varnishkafka Delivery Errors per minute on cp4017 is OK Less than 1.00% above the threshold [0.0] [08:51:30] RECOVERY - Varnishkafka Delivery Errors per minute on cp4002 is OK Less than 1.00% above the threshold [0.0] [08:51:49] RECOVERY - Varnishkafka Delivery Errors per minute on cp4006 is OK Less than 1.00% above the threshold [0.0] [08:52:08] RECOVERY - Varnishkafka Delivery Errors per minute on cp4005 is OK Less than 1.00% above the threshold [0.0] [08:52:30] <_joe_> godog: likely so, yes [08:53:09] PROBLEM - Varnishkafka Delivery Errors per minute on cp4010 is CRITICAL 11.11% of data above the critical threshold [20000.0] [08:57:23] (03PS1) 10Filippo Giunchedi: depool ulsfo due to traffic issues [dns] - 10https://gerrit.wikimedia.org/r/211681 [08:57:40] RECOVERY - Varnishkafka Delivery Errors per minute on cp4013 is OK Less than 1.00% above the threshold [0.0] [08:58:59] _joe_ akosiaris ^ thoughts ? I think we should depool [08:59:53] <_joe_> still showing packet loss? [09:00:09] PROBLEM - RAID on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:00:09] PROBLEM - puppet last run on mw1107 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:00:30] eqiad-ulsfo yeah, smokeping sees loss [09:00:38] PROBLEM - Router interfaces on cr2-ulsfo is CRITICAL No response from remote host 198.35.26.193 for 1.3.6.1.2.1.2.2.1.8 with snmp version 2 [09:00:49] PROBLEM - Varnishkafka Delivery Errors per minute on cp4014 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:00:51] <_joe_> yeah we should [09:00:58] yep, depooling [09:01:07] 6operations, 6Phabricator, 5Patch-For-Review: m3 set max_allowed_packet to 33554432 or greater - https://phabricator.wikimedia.org/T98339#1292901 (10jcrespo) @chasemp Changes has been applied to m3, unless the application maintains persistent connections/pool, it should already fix the issue, can you check i... [09:01:10] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] depool ulsfo due to traffic issues [dns] - 10https://gerrit.wikimedia.org/r/211681 (owner: 10Filippo Giunchedi) [09:01:39] PROBLEM - Varnishkafka Delivery Errors per minute on cp4002 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:01:49] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 13.33% of data above the critical threshold [500.0] [09:01:59] PROBLEM - Varnishkafka Delivery Errors per minute on cp4006 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:02:03] !log loss on ulsfo-eqiad, depooled ulsfo [09:02:08] Logged the message, Master [09:02:09] RECOVERY - Router interfaces on cr2-ulsfo is OK host 198.35.26.193, interfaces up: 77, down: 0, dormant: 0, excluded: 0, unused: 0 [09:02:09] PROBLEM - Varnishkafka Delivery Errors per minute on cp4005 is CRITICAL 22.22% of data above the critical threshold [20000.0] [09:02:29] PROBLEM - Varnishkafka Delivery Errors per minute on cp4004 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:02:29] PROBLEM - Varnishkafka Delivery Errors per minute on cp4003 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:02:38] PROBLEM - Varnishkafka Delivery Errors per minute on cp4001 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:02:49] PROBLEM - Varnishkafka Delivery Errors per minute on cp4017 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:02:49] PROBLEM - Varnishkafka Delivery Errors per minute on cp4007 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:03:08] PROBLEM - Varnishkafka Delivery Errors per minute on cp4015 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:03:29] RECOVERY - RAID on mw1107 is OK no RAID installed [09:03:29] RECOVERY - puppet last run on mw1107 is OK Puppet is currently enabled, last run 53 minutes ago with 0 failures [09:03:46] <_joe_> I'll lookl at mw1107 in a few [09:04:33] 6operations, 6Phabricator, 5Patch-For-Review: m3 set max_allowed_packet to 33554432 or greater - https://phabricator.wikimedia.org/T98339#1292907 (10mmodell) @jcrespo: phabricator no longer complains about max_allowed_packet, so it seems to be resolved. It does still complain about php post_max_size, but tha... [09:04:38] PROBLEM - Varnishkafka Delivery Errors per minute on cp4013 is CRITICAL 11.11% of data above the critical threshold [20000.0] [09:05:19] RECOVERY - Varnishkafka Delivery Errors per minute on cp4006 is OK Less than 1.00% above the threshold [0.0] [09:05:49] RECOVERY - Varnishkafka Delivery Errors per minute on cp4014 is OK Less than 1.00% above the threshold [0.0] [09:06:09] RECOVERY - Varnishkafka Delivery Errors per minute on cp4017 is OK Less than 1.00% above the threshold [0.0] [09:06:09] RECOVERY - Varnishkafka Delivery Errors per minute on cp4007 is OK Less than 1.00% above the threshold [0.0] [09:06:29] RECOVERY - Varnishkafka Delivery Errors per minute on cp4015 is OK Less than 1.00% above the threshold [0.0] [09:06:39] RECOVERY - Varnishkafka Delivery Errors per minute on cp4002 is OK Less than 1.00% above the threshold [0.0] [09:06:39] RECOVERY - Varnishkafka Delivery Errors per minute on cp4010 is OK Less than 1.00% above the threshold [0.0] [09:07:10] RECOVERY - Varnishkafka Delivery Errors per minute on cp4005 is OK Less than 1.00% above the threshold [0.0] [09:07:28]