[00:41:11] PROBLEM - NTP on furud is CRITICAL: NTP CRITICAL: No response from NTP server [00:41:12] PROBLEM - SSH on furud is CRITICAL: Server answer [00:45:02] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [00:58:21] PROBLEM - SSH on furud is CRITICAL: Server answer [01:00:32] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [01:26:49] PROBLEM - puppet last run on maerlant is CRITICAL: CRITICAL: puppet fail [01:46:43] PROBLEM - SSH on furud is CRITICAL: Server answer [01:54:23] RECOVERY - puppet last run on maerlant is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:01:42] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [02:08:21] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 653 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5297708 keys - replication_delay is 653 [02:16:11] PROBLEM - SSH on furud is CRITICAL: Server answer [02:24:22] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [02:35:31] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5280343 keys - replication_delay is 0 [02:37:12] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [1000.0] [02:38:12] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [1000.0] [02:45:14] PROBLEM - SSH on furud is CRITICAL: Server answer [02:49:22] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [02:50:41] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:52:02] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 10.192.48.44 on port 6479 [02:53:41] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:54:02] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5284640 keys - replication_delay is 0 [02:56:04] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: puppet fail [02:57:22] PROBLEM - SSH on furud is CRITICAL: Server answer [02:59:31] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [03:09:42] PROBLEM - SSH on furud is CRITICAL: Server answer [03:15:33] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [03:21:52] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:23:52] PROBLEM - YARN NodeManager Node-State on analytics1039 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:25:42] RECOVERY - YARN NodeManager Node-State on analytics1039 is OK: OK: YARN NodeManager analytics1039.eqiad.wmnet:8041 Node-State: RUNNING [03:34:02] PROBLEM - puppet last run on mw2038 is CRITICAL: CRITICAL: puppet fail [03:40:22] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 1 failures [03:47:42] PROBLEM - SSH on furud is CRITICAL: Server answer [03:51:22] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [03:57:02] PROBLEM - SSH on furud is CRITICAL: Server answer [04:02:32] RECOVERY - puppet last run on mw2038 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:06:42] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:11:43] PROBLEM - puppet last run on mw2135 is CRITICAL: CRITICAL: puppet fail [04:12:21] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [04:17:52] PROBLEM - SSH on furud is CRITICAL: Server answer [04:21:41] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [04:27:21] PROBLEM - SSH on furud is CRITICAL: Server answer [04:29:12] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [04:40:13] RECOVERY - puppet last run on mw2135 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:40:51] PROBLEM - SSH on furud is CRITICAL: Server answer [04:44:32] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [04:55:51] PROBLEM - SSH on furud is CRITICAL: Server answer [05:07:02] PROBLEM - mailman_queue_size on fermium is CRITICAL: CRITICAL: 1 mailman queue(s) above limits (thresholds: bounces: 25 in: 25 virgin: 25) [05:08:52] RECOVERY - mailman_queue_size on fermium is OK: OK: mailman queues are below the limits. [05:09:11] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [05:31:52] PROBLEM - SSH on furud is CRITICAL: Server answer [05:33:51] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [05:39:31] PROBLEM - SSH on furud is CRITICAL: Server answer [05:41:21] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [05:46:44] PROBLEM - Hadoop Namenode - Primary on analytics1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.server.namenode.NameNode [05:48:51] PROBLEM - SSH on furud is CRITICAL: Server answer [06:00:13] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [06:02:52] PROBLEM - Outgoing network saturation on labstore1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [106250000.0] [06:06:02] PROBLEM - SSH on furud is CRITICAL: Server answer [06:13:16] RECOVERY - Hadoop Namenode - Primary on analytics1001 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.namenode.NameNode [06:27:23] RECOVERY - Outgoing network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [93750000.0] [06:28:33] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [06:30:12] PROBLEM - puppet last run on restbase2006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:22] PROBLEM - puppet last run on mw2023 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:12] PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: puppet fail [06:31:33] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 3 failures [06:31:34] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:51] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:22] PROBLEM - puppet last run on db2044 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:22] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 2 failures [06:33:02] PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:33] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:42] PROBLEM - puppet last run on mw2050 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:41] PROBLEM - SSH on furud is CRITICAL: Server answer [06:55:22] RECOVERY - puppet last run on mw2023 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:56:21] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:56:32] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:57:02] RECOVERY - puppet last run on restbase2006 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:57:03] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:57:03] RECOVERY - puppet last run on db2044 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:57:42] RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:58:03] RECOVERY - puppet last run on cp3048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:21] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:22] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:32] RECOVERY - puppet last run on mw2050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:02:15] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [07:09:04] PROBLEM - SSH on furud is CRITICAL: Server answer [07:25:55] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [07:32:31] PROBLEM - SSH on furud is CRITICAL: Server answer [07:54:30] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [08:03:47] PROBLEM - SSH on furud is CRITICAL: Server answer [08:13:07] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [08:18:38] PROBLEM - SSH on furud is CRITICAL: Server answer [08:33:25] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [08:38:43] PROBLEM - SSH on furud is CRITICAL: Server answer [08:55:54] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [09:02:20] PROBLEM - SSH on furud is CRITICAL: Server answer [09:03:39] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [09:12:59] PROBLEM - SSH on furud is CRITICAL: Server answer [09:15:39] 06Operations: Furud has flapping most of the day - https://phabricator.wikimedia.org/T134098#2254488 (10Peachey88) [10:29:44] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [10:38:39] PROBLEM - SSH on furud is CRITICAL: Server answer [10:58:20] RECOVERY - Disk space on furud is OK: DISK OK [10:58:21] RECOVERY - configured eth on furud is OK: OK - interfaces up [10:58:41] RECOVERY - dhclient process on furud is OK: PROCS OK: 0 processes with command name dhclient [10:58:45] !log reboot furud.codfw.wmnet, ganeti instance with increasing load and 100% iowait, kvm/ganeti idle instance bug likely T134098 [10:58:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:59:00] RECOVERY - salt-minion processes on furud is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [10:59:19] RECOVERY - Check size of conntrack table on furud is OK: OK: nf_conntrack is 0 % full [10:59:39] RECOVERY - DPKG on furud is OK: All packages OK [10:59:49] RECOVERY - RAID on furud is OK: OK: no RAID installed [11:00:00] RECOVERY - SSH on furud is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [11:00:39] RECOVERY - puppet last run on furud is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:17:25] RECOVERY - NTP on furud is OK: NTP OK: Offset 0.001899242401 secs [11:28:33] bd808: looks like stashbot might need a little nudge [11:40:39] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] [11:43:10] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [11:45:29] PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] [11:49:20] RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [11:49:57] (03PS1) 10Volans: [TEST]: Allow to set additional grains for salt [puppet] - 10https://gerrit.wikimedia.org/r/286303 [11:51:27] (03CR) 10jenkins-bot: [V: 04-1] [TEST]: Allow to set additional grains for salt [puppet] - 10https://gerrit.wikimedia.org/r/286303 (owner: 10Volans) [12:00:11] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [12:02:15] (03PS2) 10Volans: [TEST]: Allow to set additional grains for salt [puppet] - 10https://gerrit.wikimedia.org/r/286303 [12:10:49] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:12:01] PROBLEM - RAID on sinistra is CRITICAL: CRITICAL: Active: 6, Working: 6, Failed: 2, Spare: 0 [12:19:44] 06Operations: Furud has flapping most of the day - https://phabricator.wikimedia.org/T134098#2254616 (10Peachey88) 05Open>03Resolved a:03Peachey88 > [20:58:41] !log reboot furud.codfw.wmnet, ganeti instance with increasing load and 100% iowait, kvm/ganeti idle instance bug likely T134098 > [20:58:4... [12:20:20] 06Operations: Furud has flapping most of the day - https://phabricator.wikimedia.org/T134098#2254620 (10Peachey88) a:05Peachey88>03None [12:20:53] ugh Phab is auto claiming on closures ago [12:21:43] 06Operations: Furud has flapping most of the day - https://phabricator.wikimedia.org/T134098#2254488 (10Peachey88) a:03fgiunchedi [12:34:00] PROBLEM - RAID on db1040 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:59] RECOVERY - RAID on db1040 is OK: OK: optimal, 1 logical, 2 physical [12:57:40] PROBLEM - puppet last run on mw2021 is CRITICAL: CRITICAL: puppet fail [13:26:59] RECOVERY - puppet last run on mw2021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:05:48] PROBLEM - MySQL Recent Restart on db1040 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:07:47] RECOVERY - MySQL Recent Restart on db1040 is OK: OK 72359912 seconds since restart [14:07:57] false positive heh [14:13:20] godog: ah, I'll ignore it then :) [14:14:45] godog: already checked? [14:14:57] I'll take a look anyway [14:15:11] volans: yeah the alarm itself seems false positive, though db1040 has increased load since this morning [14:15:39] YuviPanda: ^ [14:17:27] ok, thanks I'm taking a look [14:20:20] being vslow/dump it could be that was a dump [14:21:53] and there are WikiExporter::dumpFrom running, although the load is already back to normal value right now [14:26:13] ack [14:36:01] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: puppet fail [14:36:09] PROBLEM - puppet last run on graphite1001 is CRITICAL: CRITICAL: puppet fail [14:36:20] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: puppet fail [14:36:30] PROBLEM - puppet last run on wtp2013 is CRITICAL: CRITICAL: puppet fail [14:36:31] PROBLEM - puppet last run on es2011 is CRITICAL: CRITICAL: puppet fail [14:36:40] PROBLEM - puppet last run on lvs1010 is CRITICAL: CRITICAL: puppet fail [14:36:40] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: puppet fail [14:36:41] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: puppet fail [14:36:49] PROBLEM - puppet last run on dbstore2001 is CRITICAL: CRITICAL: puppet fail [14:36:59] PROBLEM - puppet last run on es1019 is CRITICAL: CRITICAL: puppet fail [14:36:59] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [14:37:00] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: puppet fail [14:37:00] PROBLEM - puppet last run on cp3049 is CRITICAL: CRITICAL: puppet fail [14:37:00] PROBLEM - puppet last run on lvs1008 is CRITICAL: CRITICAL: puppet fail [14:37:00] PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: puppet fail [14:37:00] PROBLEM - puppet last run on mw2115 is CRITICAL: CRITICAL: puppet fail [14:37:00] PROBLEM - puppet last run on elastic2018 is CRITICAL: CRITICAL: puppet fail [14:37:10] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: puppet fail [14:37:10] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: puppet fail [14:37:10] PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: puppet fail [14:37:10] PROBLEM - puppet last run on mw2071 is CRITICAL: CRITICAL: puppet fail [14:37:10] PROBLEM - puppet last run on analytics1055 is CRITICAL: CRITICAL: puppet fail [14:37:11] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: puppet fail [14:37:20] PROBLEM - puppet last run on mw2154 is CRITICAL: CRITICAL: puppet fail [14:37:20] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: puppet fail [14:37:21] PROBLEM - puppet last run on db2053 is CRITICAL: CRITICAL: puppet fail [14:37:21] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [14:37:29] PROBLEM - puppet last run on elastic2015 is CRITICAL: CRITICAL: puppet fail [14:37:29] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: puppet fail [14:37:29] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: puppet fail [14:37:30] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: puppet fail [14:37:30] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: puppet fail [14:37:39] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: puppet fail [14:37:40] PROBLEM - puppet last run on mw2108 is CRITICAL: CRITICAL: puppet fail [14:37:40] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: puppet fail [14:37:40] PROBLEM - puppet last run on mw2007 is CRITICAL: CRITICAL: puppet fail [14:37:40] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: puppet fail [14:37:40] PROBLEM - puppet last run on mw2213 is CRITICAL: CRITICAL: puppet fail [14:37:41] PROBLEM - puppet last run on restbase-test2001 is CRITICAL: CRITICAL: puppet fail [14:37:41] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: puppet fail [14:37:49] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: puppet fail [14:37:50] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: puppet fail [14:37:50] PROBLEM - puppet last run on mw2185 is CRITICAL: CRITICAL: puppet fail [14:37:50] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: puppet fail [14:37:50] PROBLEM - puppet last run on mw2013 is CRITICAL: CRITICAL: puppet fail [14:37:51] PROBLEM - puppet last run on scandium is CRITICAL: CRITICAL: puppet fail [14:37:51] godog: looks like we have puppet issues... [14:37:59] PROBLEM - puppet last run on db2030 is CRITICAL: CRITICAL: puppet fail [14:37:59] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: puppet fail [14:37:59] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [14:38:00] PROBLEM - puppet last run on mw2063 is CRITICAL: CRITICAL: puppet fail [14:38:00] PROBLEM - puppet last run on kraz is CRITICAL: CRITICAL: puppet fail [14:38:09] PROBLEM - puppet last run on labvirt1008 is CRITICAL: CRITICAL: puppet fail [14:38:10] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: puppet fail [14:38:10] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: puppet fail [14:38:20] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: puppet fail [14:38:20] PROBLEM - puppet last run on lvs1012 is CRITICAL: CRITICAL: puppet fail [14:38:20] PROBLEM - puppet last run on mc2003 is CRITICAL: CRITICAL: puppet fail [14:38:20] PROBLEM - puppet last run on mw2173 is CRITICAL: CRITICAL: puppet fail [14:38:20] PROBLEM - puppet last run on mw2066 is CRITICAL: CRITICAL: puppet fail [14:38:29] PROBLEM - puppet last run on labvirt1002 is CRITICAL: CRITICAL: puppet fail [14:38:29] PROBLEM - puppet last run on mw2180 is CRITICAL: CRITICAL: puppet fail [14:38:29] PROBLEM - puppet last run on mw2171 is CRITICAL: CRITICAL: puppet fail [14:38:29] PROBLEM - puppet last run on mw2128 is CRITICAL: CRITICAL: puppet fail [14:38:29] PROBLEM - puppet last run on wtp2018 is CRITICAL: CRITICAL: puppet fail [14:38:30] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: puppet fail [14:38:30] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: puppet fail [14:38:30] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: puppet fail [14:38:31] PROBLEM - puppet last run on logstash1004 is CRITICAL: CRITICAL: puppet fail [14:38:31] PROBLEM - puppet last run on cp2005 is CRITICAL: CRITICAL: puppet fail [14:38:32] PROBLEM - puppet last run on furud is CRITICAL: CRITICAL: puppet fail [14:38:32] PROBLEM - puppet last run on mw2206 is CRITICAL: CRITICAL: puppet fail [14:38:39] PROBLEM - puppet last run on kafka1022 is CRITICAL: CRITICAL: puppet fail [14:38:40] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: puppet fail [14:38:40] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [14:38:40] PROBLEM - puppet last run on ms-be2018 is CRITICAL: CRITICAL: puppet fail [14:38:41] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: puppet fail [14:38:41] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: puppet fail [14:38:41] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: puppet fail [14:38:49] PROBLEM - puppet last run on pybal-test2003 is CRITICAL: CRITICAL: puppet fail [14:38:49] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: puppet fail [14:38:49] PROBLEM - puppet last run on dbstore2002 is CRITICAL: CRITICAL: puppet fail [14:38:49] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: puppet fail [14:38:50] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: puppet fail [14:38:50] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: puppet fail [14:38:50] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: puppet fail [14:38:50] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: puppet fail [14:38:51] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: puppet fail [14:38:59] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: puppet fail [14:39:00] PROBLEM - puppet last run on cp3037 is CRITICAL: CRITICAL: puppet fail [14:39:00] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: puppet fail [14:39:00] PROBLEM - puppet last run on db2052 is CRITICAL: CRITICAL: puppet fail [14:39:01] PROBLEM - puppet last run on mw2104 is CRITICAL: CRITICAL: puppet fail [14:39:09] PROBLEM - puppet last run on pc1005 is CRITICAL: CRITICAL: puppet fail [14:39:09] PROBLEM - puppet last run on mw2040 is CRITICAL: CRITICAL: puppet fail [14:39:09] PROBLEM - puppet last run on hassaleh is CRITICAL: CRITICAL: puppet fail [14:39:09] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: puppet fail [14:39:10] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: puppet fail [14:39:10] PROBLEM - puppet last run on mc2010 is CRITICAL: CRITICAL: puppet fail [14:39:10] PROBLEM - puppet last run on mw2001 is CRITICAL: CRITICAL: puppet fail [14:39:10] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: puppet fail [14:39:19] PROBLEM - puppet last run on mw2060 is CRITICAL: CRITICAL: puppet fail [14:39:19] PROBLEM - puppet last run on ganeti1003 is CRITICAL: CRITICAL: puppet fail [14:39:19] PROBLEM - puppet last run on mw2137 is CRITICAL: CRITICAL: puppet fail [14:39:20] PROBLEM - puppet last run on mw2195 is CRITICAL: CRITICAL: puppet fail [14:39:20] PROBLEM - puppet last run on scb2002 is CRITICAL: CRITICAL: puppet fail [14:39:29] PROBLEM - puppet last run on mw2078 is CRITICAL: CRITICAL: puppet fail [14:39:29] PROBLEM - puppet last run on mw2022 is CRITICAL: CRITICAL: puppet fail [14:39:29] PROBLEM - puppet last run on mw2097 is CRITICAL: CRITICAL: puppet fail [14:39:30] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: puppet fail [14:39:30] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: puppet fail [14:39:40] PROBLEM - puppet last run on db1078 is CRITICAL: CRITICAL: puppet fail [14:39:40] PROBLEM - puppet last run on mw2156 is CRITICAL: CRITICAL: puppet fail [14:39:40] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: puppet fail [14:39:40] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [14:39:41] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [14:39:41] PROBLEM - puppet last run on elastic2012 is CRITICAL: CRITICAL: puppet fail [14:39:59] PROBLEM - puppet last run on elastic2023 is CRITICAL: CRITICAL: puppet fail [14:39:59] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [14:39:59] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: puppet fail [14:40:00] PROBLEM - puppet last run on restbase-test2003 is CRITICAL: CRITICAL: puppet fail [14:40:01] PROBLEM - puppet last run on rdb1008 is CRITICAL: CRITICAL: puppet fail [14:40:09] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: puppet fail [14:40:09] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: puppet fail [14:40:10] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: puppet fail [14:40:10] PROBLEM - puppet last run on mw2211 is CRITICAL: CRITICAL: puppet fail [14:40:10] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [14:40:10] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: puppet fail [14:40:10] PROBLEM - puppet last run on wtp2012 is CRITICAL: CRITICAL: puppet fail [14:40:11] PROBLEM - puppet last run on restbase2001 is CRITICAL: CRITICAL: puppet fail [14:40:11] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [14:40:12] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: puppet fail [14:40:12] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: puppet fail [14:40:19] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: puppet fail [14:40:19] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: puppet fail [14:40:20] PROBLEM - puppet last run on restbase1013 is CRITICAL: CRITICAL: puppet fail [14:40:20] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: puppet fail [14:40:20] PROBLEM - puppet last run on labvirt1003 is CRITICAL: CRITICAL: puppet fail [14:40:21] PROBLEM - puppet last run on mw2017 is CRITICAL: CRITICAL: puppet fail [14:40:29] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: puppet fail [14:40:29] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: puppet fail [14:40:29] PROBLEM - puppet last run on wtp2011 is CRITICAL: CRITICAL: puppet fail [14:40:29] PROBLEM - puppet last run on labvirt1006 is CRITICAL: CRITICAL: puppet fail [14:40:30] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [14:40:30] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: puppet fail [14:40:30] PROBLEM - puppet last run on mw2011 is CRITICAL: CRITICAL: puppet fail [14:40:30] PROBLEM - puppet last run on cp2008 is CRITICAL: CRITICAL: puppet fail [14:40:31] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: puppet fail [14:40:31] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: puppet fail [14:40:39] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: puppet fail [14:40:39] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [14:40:40] PROBLEM - puppet last run on mw2178 is CRITICAL: CRITICAL: puppet fail [14:40:49] PROBLEM - puppet last run on rdb2001 is CRITICAL: CRITICAL: puppet fail [14:40:49] PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: puppet fail [14:40:49] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: puppet fail [14:40:49] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: puppet fail [14:40:50] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: puppet fail [14:41:00] PROBLEM - puppet last run on es1011 is CRITICAL: CRITICAL: puppet fail [14:41:00] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: puppet fail [14:41:00] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail [14:41:01] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: puppet fail [14:41:01] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [14:41:09] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [14:41:09] PROBLEM - puppet last run on mw2119 is CRITICAL: CRITICAL: puppet fail [14:41:09] PROBLEM - puppet last run on cp3038 is CRITICAL: CRITICAL: puppet fail [14:41:09] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [14:41:09] PROBLEM - puppet last run on mc2011 is CRITICAL: CRITICAL: puppet fail [14:41:09] PROBLEM - puppet last run on mc2009 is CRITICAL: CRITICAL: puppet fail [14:41:10] PROBLEM - puppet last run on cp3041 is CRITICAL: CRITICAL: puppet fail [14:41:11] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: puppet fail [14:41:11] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: puppet fail [14:41:19] PROBLEM - puppet last run on mw2002 is CRITICAL: CRITICAL: puppet fail [14:41:19] PROBLEM - puppet last run on mw2059 is CRITICAL: CRITICAL: puppet fail [14:41:20] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: puppet fail [14:41:20] PROBLEM - puppet last run on elastic2003 is CRITICAL: CRITICAL: puppet fail [14:41:29] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: puppet fail [14:41:29] PROBLEM - puppet last run on conf1002 is CRITICAL: CRITICAL: puppet fail [14:41:29] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [14:41:39] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: puppet fail [14:41:39] PROBLEM - puppet last run on mw2029 is CRITICAL: CRITICAL: puppet fail [14:41:39] PROBLEM - puppet last run on mw2147 is CRITICAL: CRITICAL: puppet fail [14:41:40] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: puppet fail [14:41:51] PROBLEM - puppet last run on logstash1006 is CRITICAL: CRITICAL: puppet fail [14:41:51] PROBLEM - puppet last run on db1075 is CRITICAL: CRITICAL: puppet fail [14:41:52] PROBLEM - puppet last run on db2043 is CRITICAL: CRITICAL: puppet fail [14:41:52] PROBLEM - puppet last run on restbase1015 is CRITICAL: CRITICAL: puppet fail [14:41:52] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: puppet fail [14:42:00] PROBLEM - puppet last run on mw2086 is CRITICAL: CRITICAL: puppet fail [14:42:00] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: puppet fail [14:42:00] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [14:42:00] PROBLEM - puppet last run on db2042 is CRITICAL: CRITICAL: puppet fail [14:42:01] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [14:42:01] PROBLEM - puppet last run on mw2190 is CRITICAL: CRITICAL: puppet fail [14:42:01] PROBLEM - puppet last run on conf1003 is CRITICAL: CRITICAL: puppet fail [14:42:10] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [14:42:10] PROBLEM - puppet last run on cp3044 is CRITICAL: CRITICAL: puppet fail [14:42:19] PROBLEM - puppet last run on mw2192 is CRITICAL: CRITICAL: puppet fail [14:42:19] PROBLEM - puppet last run on mw2025 is CRITICAL: CRITICAL: puppet fail [14:42:20] PROBLEM - puppet last run on scb1002 is CRITICAL: CRITICAL: puppet fail [14:42:20] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: puppet fail [14:42:21] PROBLEM - puppet last run on mw2165 is CRITICAL: CRITICAL: puppet fail [14:42:21] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [14:42:29] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: puppet fail [14:42:29] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: puppet fail [14:42:30] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: puppet fail [14:42:30] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: puppet fail [14:42:30] PROBLEM - puppet last run on es1012 is CRITICAL: CRITICAL: puppet fail [14:42:30] PROBLEM - puppet last run on es1015 is CRITICAL: CRITICAL: puppet fail [14:42:30] PROBLEM - puppet last run on labvirt1001 is CRITICAL: CRITICAL: puppet fail [14:42:31] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: puppet fail [14:42:31] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: puppet fail [14:42:33] O_O [14:42:39] PROBLEM - puppet last run on elastic2016 is CRITICAL: CRITICAL: puppet fail [14:42:39] PROBLEM - puppet last run on cp2012 is CRITICAL: CRITICAL: puppet fail [14:42:40] PROBLEM - puppet last run on cp2020 is CRITICAL: CRITICAL: puppet fail [14:42:41] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: puppet fail [14:42:41] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: puppet fail [14:42:49] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [14:42:50] PROBLEM - puppet last run on elastic2011 is CRITICAL: CRITICAL: puppet fail [14:43:00] PROBLEM - puppet last run on cp3043 is CRITICAL: CRITICAL: puppet fail [14:43:00] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: puppet fail [14:43:00] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: puppet fail [14:43:00] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: puppet fail [14:43:00] PROBLEM - puppet last run on mw2135 is CRITICAL: CRITICAL: puppet fail [14:43:07] * Platonides suspects someone is getting called by this [14:43:09] PROBLEM - puppet last run on ms-be2019 is CRITICAL: CRITICAL: puppet fail [14:43:10] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: puppet fail [14:43:10] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: puppet fail [14:43:21] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: puppet fail [14:43:21] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: puppet fail [14:43:21] PROBLEM - puppet last run on cp3033 is CRITICAL: CRITICAL: puppet fail [14:43:30] PROBLEM - puppet last run on cp1074 is CRITICAL: CRITICAL: puppet fail [14:43:30] PROBLEM - puppet last run on mw2194 is CRITICAL: CRITICAL: puppet fail [14:43:30] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: puppet fail [14:43:30] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: puppet fail [14:43:30] PROBLEM - puppet last run on mw2149 is CRITICAL: CRITICAL: puppet fail [14:43:31] PROBLEM - puppet last run on mw2006 is CRITICAL: CRITICAL: puppet fail [14:43:31] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: puppet fail [14:43:31] PROBLEM - puppet last run on mw2179 is CRITICAL: CRITICAL: puppet fail [14:43:31] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: puppet fail [14:43:32] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: puppet fail [14:43:39] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail [14:43:40] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: puppet fail [14:43:41] PROBLEM - puppet last run on ms-be1016 is CRITICAL: CRITICAL: puppet fail [14:43:50] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: puppet fail [14:43:50] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: puppet fail [14:43:51] PROBLEM - puppet last run on oresrdb1002 is CRITICAL: CRITICAL: puppet fail [14:43:59] PROBLEM - puppet last run on analytics1049 is CRITICAL: CRITICAL: puppet fail [14:43:59] PROBLEM - puppet last run on mw2102 is CRITICAL: CRITICAL: puppet fail [14:43:59] PROBLEM - puppet last run on mw2005 is CRITICAL: CRITICAL: puppet fail [14:43:59] PROBLEM - puppet last run on es1013 is CRITICAL: CRITICAL: puppet fail [14:44:00] PROBLEM - puppet last run on analytics1046 is CRITICAL: CRITICAL: puppet fail [14:44:00] PROBLEM - puppet last run on mw2042 is CRITICAL: CRITICAL: puppet fail [14:44:00] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: puppet fail [14:44:00] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: puppet fail [14:44:00] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: puppet fail [14:44:01] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: puppet fail [14:44:01] PROBLEM - puppet last run on cp3035 is CRITICAL: CRITICAL: puppet fail [14:44:02] PROBLEM - puppet last run on es1017 is CRITICAL: CRITICAL: puppet fail [14:44:08] YuviPanda: are you still around? [14:44:10] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: puppet fail [14:44:10] PROBLEM - puppet last run on mw2151 is CRITICAL: CRITICAL: puppet fail [14:44:10] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: puppet fail [14:44:11] PROBLEM - puppet last run on db2048 is CRITICAL: CRITICAL: puppet fail [14:44:19] PROBLEM - puppet last run on mw2065 is CRITICAL: CRITICAL: puppet fail [14:44:19] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: puppet fail [14:44:20] PROBLEM - puppet last run on mw2183 is CRITICAL: CRITICAL: puppet fail [14:44:20] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: puppet fail [14:44:20] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: puppet fail [14:44:20] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: puppet fail [14:44:21] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: puppet fail [14:44:29] PROBLEM - puppet last run on mw2099 is CRITICAL: CRITICAL: puppet fail [14:44:29] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: puppet fail [14:44:30] PROBLEM - puppet last run on es1018 is CRITICAL: CRITICAL: puppet fail [14:44:30] PROBLEM - puppet last run on mw2197 is CRITICAL: CRITICAL: puppet fail [14:44:30] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: puppet fail [14:44:30] PROBLEM - puppet last run on maps-test2002 is CRITICAL: CRITICAL: puppet fail [14:44:30] PROBLEM - puppet last run on cp2021 is CRITICAL: CRITICAL: puppet fail [14:44:30] PROBLEM - puppet last run on mw2089 is CRITICAL: CRITICAL: puppet fail [14:44:39] PROBLEM - puppet last run on cp2018 is CRITICAL: CRITICAL: puppet fail [14:44:45] !log truncated puppet.fact_values table to fix puppet (as documented on wikitech) [14:44:49] PROBLEM - puppet last run on analytics1001 is CRITICAL: CRITICAL: puppet fail [14:44:49] PROBLEM - puppet last run on elastic2009 is CRITICAL: CRITICAL: puppet fail [14:44:49] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: puppet fail [14:44:50] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: puppet fail [14:44:51] PROBLEM - puppet last run on dbproxy1004 is CRITICAL: CRITICAL: puppet fail [14:44:51] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: puppet fail [14:44:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:44:59] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: puppet fail [14:44:59] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: puppet fail [14:45:00] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: puppet fail [14:45:00] PROBLEM - puppet last run on restbase1010 is CRITICAL: CRITICAL: puppet fail [14:45:00] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: puppet fail [14:45:00] PROBLEM - puppet last run on wtp2014 is CRITICAL: CRITICAL: puppet fail [14:45:00] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [14:45:00] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: puppet fail [14:45:01] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [14:45:01] PROBLEM - puppet last run on ganeti2005 is CRITICAL: CRITICAL: puppet fail [14:45:09] PROBLEM - puppet last run on labvirt1011 is CRITICAL: CRITICAL: puppet fail [14:45:10] PROBLEM - puppet last run on mw2044 is CRITICAL: CRITICAL: puppet fail [14:45:19] PROBLEM - puppet last run on mw2112 is CRITICAL: CRITICAL: puppet fail [14:45:19] PROBLEM - puppet last run on mw2140 is CRITICAL: CRITICAL: puppet fail [14:45:19] PROBLEM - puppet last run on wtp2007 is CRITICAL: CRITICAL: puppet fail [14:45:20] PROBLEM - puppet last run on pc2004 is CRITICAL: CRITICAL: puppet fail [14:45:20] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [14:45:20] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: puppet fail [14:45:21] PROBLEM - puppet last run on mw2041 is CRITICAL: CRITICAL: puppet fail [14:45:29] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: puppet fail [14:45:29] PROBLEM - puppet last run on cp1072 is CRITICAL: CRITICAL: puppet fail [14:45:40] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: puppet fail [14:45:40] PROBLEM - puppet last run on restbase2005 is CRITICAL: CRITICAL: puppet fail [14:45:40] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: puppet fail [14:45:40] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: puppet fail [14:45:40] PROBLEM - puppet last run on mw2189 is CRITICAL: CRITICAL: puppet fail [14:45:40] PROBLEM - puppet last run on eventlog1001 is CRITICAL: CRITICAL: puppet fail [14:45:41] PROBLEM - puppet last run on elastic2002 is CRITICAL: CRITICAL: puppet fail [14:45:41] PROBLEM - puppet last run on rdb2002 is CRITICAL: CRITICAL: puppet fail [14:45:41] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: puppet fail [14:45:42] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: puppet fail [14:45:49] PROBLEM - puppet last run on mw2064 is CRITICAL: CRITICAL: puppet fail [14:45:50] PROBLEM - puppet last run on mw2170 is CRITICAL: CRITICAL: puppet fail [14:45:50] PROBLEM - puppet last run on db2046 is CRITICAL: CRITICAL: puppet fail [14:45:50] PROBLEM - puppet last run on wmf4727-test is CRITICAL: CRITICAL: puppet fail [14:45:50] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [14:45:59] PROBLEM - puppet last run on mw2160 is CRITICAL: CRITICAL: puppet fail [14:45:59] PROBLEM - puppet last run on mw2103 is CRITICAL: CRITICAL: puppet fail [14:45:59] PROBLEM - puppet last run on mw2100 is CRITICAL: CRITICAL: puppet fail [14:45:59] PROBLEM - puppet last run on cp2007 is CRITICAL: CRITICAL: puppet fail [14:46:00] PROBLEM - puppet last run on db2069 is CRITICAL: CRITICAL: puppet fail [14:46:00] PROBLEM - puppet last run on aqs1003 is CRITICAL: CRITICAL: puppet fail [14:46:00] PROBLEM - puppet last run on analytics1043 is CRITICAL: CRITICAL: puppet fail [14:46:00] PROBLEM - puppet last run on mw2116 is CRITICAL: CRITICAL: puppet fail [14:46:01] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail [14:46:01] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: puppet fail [14:46:09] PROBLEM - puppet last run on mx2001 is CRITICAL: CRITICAL: puppet fail [14:46:09] PROBLEM - puppet last run on cp2025 is CRITICAL: CRITICAL: puppet fail [14:46:09] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: puppet fail [14:46:10] PROBLEM - puppet last run on cp3034 is CRITICAL: CRITICAL: puppet fail [14:46:11] PROBLEM - puppet last run on elastic2022 is CRITICAL: CRITICAL: puppet fail [14:46:11] PROBLEM - puppet last run on californium is CRITICAL: CRITICAL: puppet fail [14:46:19] PROBLEM - puppet last run on db2070 is CRITICAL: CRITICAL: puppet fail [14:46:19] PROBLEM - puppet last run on mc2002 is CRITICAL: CRITICAL: puppet fail [14:46:19] PROBLEM - puppet last run on ms-be1018 is CRITICAL: CRITICAL: puppet fail [14:46:19] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: puppet fail [14:46:20] PROBLEM - puppet last run on mw2139 is CRITICAL: CRITICAL: puppet fail [14:46:20] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: puppet fail [14:46:20] PROBLEM - puppet last run on mw2161 is CRITICAL: CRITICAL: puppet fail [14:46:20] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: puppet fail [14:46:21] PROBLEM - puppet last run on restbase1011 is CRITICAL: CRITICAL: puppet fail [14:46:21] PROBLEM - puppet last run on wtp2020 is CRITICAL: CRITICAL: puppet fail [14:46:22] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: puppet fail [14:46:29] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: puppet fail [14:46:30] PROBLEM - puppet last run on elastic2019 is CRITICAL: CRITICAL: puppet fail [14:46:30] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: puppet fail [14:46:30] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: puppet fail [14:46:30] PROBLEM - puppet last run on wtp2002 is CRITICAL: CRITICAL: puppet fail [14:46:30] PROBLEM - puppet last run on mw2031 is CRITICAL: CRITICAL: puppet fail [14:46:30] PROBLEM - puppet last run on mw2153 is CRITICAL: CRITICAL: puppet fail [14:46:31] PROBLEM - puppet last run on mw2201 is CRITICAL: CRITICAL: puppet fail [14:46:39] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: puppet fail [14:46:40] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: puppet fail [14:46:40] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: puppet fail [14:46:49] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: puppet fail [14:46:50] PROBLEM - puppet last run on kafka1001 is CRITICAL: CRITICAL: puppet fail [14:46:50] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: puppet fail [14:46:50] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: puppet fail [14:46:51] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: puppet fail [14:47:00] PROBLEM - puppet last run on mw2009 is CRITICAL: CRITICAL: puppet fail [14:47:00] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: puppet fail [14:47:09] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [14:47:10] PROBLEM - puppet last run on mw2164 is CRITICAL: CRITICAL: puppet fail [14:47:10] PROBLEM - puppet last run on mw2015 is CRITICAL: CRITICAL: puppet fail [14:47:10] PROBLEM - puppet last run on mw2105 is CRITICAL: CRITICAL: puppet fail [14:47:10] PROBLEM - puppet last run on elastic2017 is CRITICAL: CRITICAL: puppet fail [14:47:19] PROBLEM - puppet last run on pybal-test2001 is CRITICAL: CRITICAL: puppet fail [14:47:19] PROBLEM - puppet last run on mc2012 is CRITICAL: CRITICAL: puppet fail [14:47:20] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: puppet fail [14:47:29] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: puppet fail [14:47:29] PROBLEM - puppet last run on analytics1048 is CRITICAL: CRITICAL: puppet fail [14:47:29] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: puppet fail [14:47:29] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: puppet fail [14:47:30] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: puppet fail [14:47:30] PROBLEM - puppet last run on auth1001 is CRITICAL: CRITICAL: puppet fail [14:47:30] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: puppet fail [14:47:30] PROBLEM - puppet last run on mw2109 is CRITICAL: CRITICAL: puppet fail [14:47:31] PROBLEM - puppet last run on ganeti1002 is CRITICAL: CRITICAL: puppet fail [14:47:31] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: puppet fail [14:47:32] PROBLEM - puppet last run on restbase-test2002 is CRITICAL: CRITICAL: puppet fail [14:47:39] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: puppet fail [14:47:39] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: puppet fail [14:47:39] PROBLEM - puppet last run on ganeti1004 is CRITICAL: CRITICAL: puppet fail [14:47:40] PROBLEM - puppet last run on mw2199 is CRITICAL: CRITICAL: puppet fail [14:47:40] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: puppet fail [14:47:40] PROBLEM - puppet last run on mw2004 is CRITICAL: CRITICAL: puppet fail [14:47:49] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: puppet fail [14:47:50] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: puppet fail [14:47:50] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: puppet fail [14:47:50] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: puppet fail [14:47:50] PROBLEM - puppet last run on mw2117 is CRITICAL: CRITICAL: puppet fail [14:47:59] PROBLEM - puppet last run on analytics1057 is CRITICAL: CRITICAL: puppet fail [14:47:59] PROBLEM - puppet last run on mw2075 is CRITICAL: CRITICAL: puppet fail [14:48:00] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: puppet fail [14:48:01] PROBLEM - puppet last run on lvs1007 is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on ganeti2001 is CRITICAL: CRITICAL: puppet fail [14:48:10] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: puppet fail [14:48:20] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: puppet fail [14:48:20] PROBLEM - puppet last run on mw2083 is CRITICAL: CRITICAL: puppet fail [14:48:21] PROBLEM - puppet last run on mw2188 is CRITICAL: CRITICAL: puppet fail [14:48:21] PROBLEM - puppet last run on db2008 is CRITICAL: CRITICAL: puppet fail [14:48:29] PROBLEM - puppet last run on mw2010 is CRITICAL: CRITICAL: puppet fail [14:48:30] PROBLEM - puppet last run on wtp2001 is CRITICAL: CRITICAL: puppet fail [14:48:30] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:48:30] PROBLEM - puppet last run on mw2080 is CRITICAL: CRITICAL: puppet fail [14:48:30] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: puppet fail [14:48:31] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: puppet fail [14:48:31] PROBLEM - puppet last run on cp2023 is CRITICAL: CRITICAL: puppet fail [14:48:40] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: puppet fail [14:48:40] PROBLEM - puppet last run on cp2026 is CRITICAL: CRITICAL: puppet fail [14:48:40] PROBLEM - puppet last run on db2045 is CRITICAL: CRITICAL: puppet fail [14:48:40] PROBLEM - puppet last run on mw2019 is CRITICAL: CRITICAL: puppet fail [14:48:40] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: puppet fail [14:48:41] PROBLEM - puppet last run on planet1001 is CRITICAL: CRITICAL: puppet fail [14:48:41] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: puppet fail [14:48:41] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: puppet fail [14:48:49] PROBLEM - puppet last run on mw2082 is CRITICAL: CRITICAL: puppet fail [14:48:50] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [14:48:50] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: puppet fail [14:48:50] PROBLEM - puppet last run on wtp2005 is CRITICAL: CRITICAL: puppet fail [14:48:50] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: puppet fail [14:48:50] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: puppet fail [14:48:51] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [14:48:59] PROBLEM - puppet last run on mc2016 is CRITICAL: CRITICAL: puppet fail [14:49:00] PROBLEM - puppet last run on mw2087 is CRITICAL: CRITICAL: puppet fail [14:49:00] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: puppet fail [14:49:00] PROBLEM - puppet last run on ganeti2004 is CRITICAL: CRITICAL: puppet fail [14:49:10] PROBLEM - puppet last run on mw2163 is CRITICAL: CRITICAL: puppet fail [14:49:11] PROBLEM - puppet last run on rdb2006 is CRITICAL: CRITICAL: puppet fail [14:49:11] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: puppet fail [14:49:20] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: puppet fail [14:49:20] PROBLEM - puppet last run on cp3040 is CRITICAL: CRITICAL: puppet fail [14:49:20] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: puppet fail [14:49:29] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: puppet fail [14:49:29] PROBLEM - puppet last run on db2059 is CRITICAL: CRITICAL: puppet fail [14:49:29] PROBLEM - puppet last run on mw2114 is CRITICAL: CRITICAL: puppet fail [14:49:30] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: puppet fail [14:49:49] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: puppet fail [14:49:50] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: puppet fail [14:49:50] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: puppet fail [14:49:50] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: puppet fail [14:50:00] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: puppet fail [14:50:00] PROBLEM - puppet last run on cp3047 is CRITICAL: CRITICAL: puppet fail [14:50:10] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:57:40] RECOVERY - puppet last run on restbase-test2003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:01:18] (03PS1) 10Volans: Depool db1040 for investigation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/286309 (https://phabricator.wikimedia.org/T134114) [15:02:44] (03CR) 10Volans: [C: 032] Depool db1040 for investigation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/286309 (https://phabricator.wikimedia.org/T134114) (owner: 10Volans) [15:02:59] RECOVERY - puppet last run on cp3049 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:03:00] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [15:03:09] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [15:03:10] (03Merged) 10jenkins-bot: Depool db1040 for investigation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/286309 (https://phabricator.wikimedia.org/T134114) (owner: 10Volans) [15:03:20] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:03:29] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:03:30] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:03:30] RECOVERY - puppet last run on mw2108 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:03:40] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:03:40] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:03:40] RECOVERY - puppet last run on restbase-test2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:03:49] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [15:03:50] RECOVERY - puppet last run on db2030 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:03:51] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:03:51] RECOVERY - puppet last run on graphite1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:00] RECOVERY - puppet last run on kraz is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:04:00] RECOVERY - puppet last run on labvirt1008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:04:01] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [15:04:09] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:04:19] RECOVERY - puppet last run on mc2003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:04:20] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:04:20] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [15:04:20] RECOVERY - puppet last run on wtp2013 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [15:04:30] RECOVERY - puppet last run on lvs1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:30] RECOVERY - puppet last run on cp2005 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:04:30] RECOVERY - puppet last run on es2011 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:04:30] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:04:31] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:04:39] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [15:04:40] RECOVERY - puppet last run on dbstore2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:49] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:04:49] RECOVERY - puppet last run on es1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:50] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:50] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:50] RECOVERY - puppet last run on lvs1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:50] RECOVERY - puppet last run on mw2148 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [15:04:59] RECOVERY - puppet last run on elastic2018 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:05:00] RECOVERY - puppet last run on pc1005 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:05:00] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:00] RECOVERY - puppet last run on analytics1055 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:05:00] RECOVERY - puppet last run on hassaleh is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:05:00] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:01] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:05:09] RECOVERY - puppet last run on mw2071 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [15:05:09] RECOVERY - puppet last run on mc2010 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [15:05:10] RECOVERY - puppet last run on ganeti1003 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:05:10] RECOVERY - puppet last run on mw2154 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:05:19] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:19] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:20] RECOVERY - puppet last run on db2053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:20] RECOVERY - puppet last run on scb2002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [15:05:20] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:20] RECOVERY - puppet last run on elastic2015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:20] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:21] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:21] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [15:05:30] RECOVERY - puppet last run on db1078 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:05:30] RECOVERY - puppet last run on mw2156 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [15:05:39] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:05:39] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:05:40] RECOVERY - puppet last run on mw2213 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:40] RECOVERY - puppet last run on elastic2012 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [15:05:49] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:05:49] RECOVERY - puppet last run on scandium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:49] RECOVERY - puppet last run on mw2185 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [15:05:50] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:50] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:50] RECOVERY - puppet last run on mw2013 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:05:50] RECOVERY - puppet last run on elastic2023 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:05:59] RECOVERY - puppet last run on mw2063 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:05:59] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:06:00] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:00] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:06:00] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:06:09] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:06:09] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:10] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:06:10] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:10] RECOVERY - puppet last run on restbase1013 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:06:11] RECOVERY - puppet last run on lvs1012 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [15:06:19] RECOVERY - puppet last run on labvirt1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:06:19] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [15:06:19] RECOVERY - puppet last run on labvirt1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:19] RECOVERY - puppet last run on mw2173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:20] RECOVERY - puppet last run on mw2066 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:06:20] RECOVERY - puppet last run on labvirt1006 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [15:06:20] RECOVERY - puppet last run on wtp2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:20] RECOVERY - puppet last run on mw2180 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:21] RECOVERY - puppet last run on mw2171 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:06:21] RECOVERY - puppet last run on mw2128 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:06:22] RECOVERY - puppet last run on wtp2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:22] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:23] RECOVERY - puppet last run on logstash1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:23] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [15:06:29] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:06:29] RECOVERY - puppet last run on kafka1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:29] RECOVERY - puppet last run on cp2008 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [15:06:30] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:30] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:06:30] RECOVERY - puppet last run on furud is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:41] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:49] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [15:06:50] RECOVERY - puppet last run on es1011 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:06:50] RECOVERY - puppet last run on mw2115 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:50] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:59] RECOVERY - puppet last run on cp3037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:59] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:06:59] RECOVERY - puppet last run on db2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:00] RECOVERY - puppet last run on mw2104 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:07:00] RECOVERY - puppet last run on mw2040 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [15:07:00] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:07:00] RECOVERY - puppet last run on cp3038 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [15:07:02] RECOVERY - puppet last run on mc2011 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:07:02] RECOVERY - puppet last run on mc2009 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [15:07:02] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:07:02] RECOVERY - puppet last run on mw2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:09] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:10] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:07:10] RECOVERY - puppet last run on mw2195 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:07:10] RECOVERY - puppet last run on mw2137 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:07:19] RECOVERY - puppet last run on elastic2003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [15:07:20] RECOVERY - puppet last run on conf1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:20] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:20] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:20] RECOVERY - puppet last run on mw2078 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:07:20] RECOVERY - puppet last run on mw2022 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:07:20] RECOVERY - puppet last run on mw2097 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:07:30] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:30] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:07:30] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:07:31] RECOVERY - puppet last run on mw2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:40] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:41] RECOVERY - puppet last run on logstash1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:41] RECOVERY - puppet last run on db1075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:48] !log volans@tin Synchronized wmf-config/db-eqiad.php: Depool db1040 for investigation T134114 (duration: 01m 22s) [15:07:49] RECOVERY - Unmerged changes on repository mediawiki_config on mira is OK: No changes to merge. [15:07:49] RECOVERY - puppet last run on restbase1015 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [15:07:49] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [15:07:49] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:50] RECOVERY - puppet last run on rdb1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:07:59] RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [15:07:59] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:59] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:07:59] RECOVERY - puppet last run on mw2211 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [15:08:00] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:08:00] RECOVERY - puppet last run on wtp2012 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:08:00] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:00] RECOVERY - puppet last run on restbase2001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:08:00] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:08:09] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:10] RECOVERY - puppet last run on scb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:11] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:19] RECOVERY - puppet last run on mw2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:20] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [15:08:21] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [15:08:21] RECOVERY - puppet last run on mw2011 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:08:21] RECOVERY - puppet last run on es1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:21] RECOVERY - puppet last run on es1012 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:08:30] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:08:30] RECOVERY - puppet last run on labvirt1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [15:08:30] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:30] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:31] RECOVERY - puppet last run on mw2206 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:31] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:31] RECOVERY - puppet last run on elastic2016 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:08:31] RECOVERY - puppet last run on cp2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:39] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:39] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:08:39] RECOVERY - puppet last run on cp2020 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [15:08:40] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:08:40] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:08:40] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:49] RECOVERY - puppet last run on elastic2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:50] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:50] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:00] RECOVERY - puppet last run on mw2119 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:09:00] RECOVERY - puppet last run on cp3043 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:09:00] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:00] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [15:09:00] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:00] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:09:00] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [15:09:01] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:09:01] RECOVERY - puppet last run on cp3041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:02] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:10] RECOVERY - puppet last run on mw2002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [15:09:10] RECOVERY - puppet last run on mw2060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:10] RECOVERY - puppet last run on mw2059 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:09:11] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:12] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:09:12] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:19] RECOVERY - puppet last run on cp3033 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:09:29] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:09:29] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:09:30] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:09:30] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [15:09:30] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [15:09:30] RECOVERY - puppet last run on mw2029 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:09:31] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:09:41] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:49] RECOVERY - puppet last run on db2043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:50] RECOVERY - puppet last run on oresrdb1002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:09:50] RECOVERY - puppet last run on analytics1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:50] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:50] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:50] RECOVERY - puppet last run on mw2086 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:51] RECOVERY - puppet last run on analytics1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:59] RECOVERY - puppet last run on conf1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:59] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [15:09:59] RECOVERY - puppet last run on mw2190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:59] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:10:00] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:00] RECOVERY - puppet last run on es1017 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:10:00] RECOVERY - puppet last run on cp3044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:01] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:10:10] RECOVERY - puppet last run on mw2192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:10] RECOVERY - puppet last run on db2048 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [15:10:10] RECOVERY - puppet last run on mw2025 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [15:10:10] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:10] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:19] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [15:10:20] RECOVERY - puppet last run on mw2165 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:10:20] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [15:10:20] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [15:10:20] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:10:21] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:21] RECOVERY - puppet last run on es1018 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:10:30] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [15:10:30] RECOVERY - puppet last run on mw2178 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:30] RECOVERY - puppet last run on mw2089 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [15:10:40] RECOVERY - puppet last run on dbproxy1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:41] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [15:10:49] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:50] RECOVERY - puppet last run on restbase1010 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:10:50] RECOVERY - puppet last run on wtp2014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:50] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:51] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:10:59] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [15:11:00] RECOVERY - puppet last run on mw2135 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:00] RECOVERY - puppet last run on ganeti2005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:11:00] RECOVERY - puppet last run on ms-be2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:10] RECOVERY - puppet last run on wtp2007 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [15:11:10] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:11] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:19] RECOVERY - puppet last run on cp1072 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [15:11:20] RECOVERY - puppet last run on cp1074 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:29] RECOVERY - puppet last run on mw2194 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:11:29] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:11:29] RECOVERY - puppet last run on mw2149 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:30] RECOVERY - puppet last run on mw2006 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:11:30] RECOVERY - puppet last run on mw2147 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:30] RECOVERY - puppet last run on restbase2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:30] RECOVERY - puppet last run on mw2189 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:11:31] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:31] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:11:31] RECOVERY - puppet last run on ms-be1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:41] RECOVERY - puppet last run on wmf4727-test is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [15:11:49] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [15:11:49] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:11:50] RECOVERY - puppet last run on mw2103 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:11:50] RECOVERY - puppet last run on cp2007 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [15:11:50] RECOVERY - puppet last run on es1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:50] RECOVERY - puppet last run on mw2102 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:11:50] RECOVERY - puppet last run on mw2005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:11:51] RECOVERY - puppet last run on db2069 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:11:51] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:00] RECOVERY - puppet last run on mw2042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:00] RECOVERY - puppet last run on mx2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:00] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:00] RECOVERY - puppet last run on cp3035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:01] RECOVERY - puppet last run on cp3034 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:12:01] RECOVERY - puppet last run on californium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [15:12:01] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:12:01] RECOVERY - puppet last run on elastic2022 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [15:12:09] RECOVERY - puppet last run on mw2151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:09] RECOVERY - puppet last run on ms-be1018 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:12:09] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [15:12:10] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [15:12:10] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:19] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:20] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:12:20] RECOVERY - puppet last run on mw2099 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:20] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:20] RECOVERY - puppet last run on elastic2019 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [15:12:21] RECOVERY - puppet last run on mw2197 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [15:12:21] RECOVERY - puppet last run on maps-test2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:21] RECOVERY - puppet last run on cp2021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:29] RECOVERY - puppet last run on cp2018 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [15:12:39] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:12:40] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:40] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [15:12:40] RECOVERY - puppet last run on elastic2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:40] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:40] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:12:41] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:12:49] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:12:50] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [15:12:50] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:12:51] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:12:51] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:12:51] RECOVERY - puppet last run on labvirt1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:00] RECOVERY - puppet last run on mw2044 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:13:00] RECOVERY - puppet last run on mw2112 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [15:13:10] RECOVERY - puppet last run on mw2140 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:10] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:11] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:13:11] RECOVERY - puppet last run on pybal-test2001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:13:11] RECOVERY - puppet last run on mc2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:11] RECOVERY - puppet last run on pc2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:11] RECOVERY - puppet last run on mw2041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:12] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:13:19] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [15:13:19] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:13:20] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:20] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:20] RECOVERY - puppet last run on eventlog1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:21] RECOVERY - puppet last run on mw2179 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:30] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [15:13:30] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:30] RECOVERY - puppet last run on elastic2002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:13:30] RECOVERY - puppet last run on restbase-test2002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [15:13:30] RECOVERY - puppet last run on ganeti1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:30] RECOVERY - puppet last run on rdb2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:39] RECOVERY - puppet last run on mw2170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:40] RECOVERY - puppet last run on db2046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:40] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:49] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [15:13:50] RECOVERY - puppet last run on mw2160 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:50] RECOVERY - puppet last run on analytics1057 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:13:50] RECOVERY - puppet last run on analytics1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:50] RECOVERY - puppet last run on aqs1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:50] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:51] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:51] RECOVERY - puppet last run on mw2116 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [15:13:51] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [15:13:52] RECOVERY - puppet last run on lvs1007 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [15:13:52] RECOVERY - puppet last run on cp2025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:01] RECOVERY - puppet last run on db2070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:01] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:14:09] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:14:09] RECOVERY - puppet last run on ganeti2001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:14:09] RECOVERY - puppet last run on mc2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:10] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:14:10] RECOVERY - puppet last run on mw2139 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:10] RECOVERY - puppet last run on mw2065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:10] RECOVERY - puppet last run on mw2161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:11] RECOVERY - puppet last run on restbase1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:11] RECOVERY - puppet last run on mw2183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:12] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:19] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:19] RECOVERY - puppet last run on wtp2020 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:14:20] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:20] RECOVERY - puppet last run on mw2188 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:14:20] RECOVERY - puppet last run on wtp2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:21] RECOVERY - puppet last run on mw2031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:29] RECOVERY - puppet last run on mw2153 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:14:29] RECOVERY - puppet last run on mw2201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:14:29] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:29] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:30] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:14:30] RECOVERY - puppet last run on cp2023 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:14:30] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [15:14:30] RECOVERY - puppet last run on cp2026 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [15:14:39] RECOVERY - puppet last run on db2045 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:14:40] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [15:14:40] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:40] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:14:40] RECOVERY - puppet last run on kafka1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:40] RECOVERY - puppet last run on planet1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:41] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:41] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [15:14:49] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:14:50] RECOVERY - puppet last run on wtp2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:51] RECOVERY - puppet last run on mc2016 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:14:51] RECOVERY - puppet last run on mw2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:14:51] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:14:51] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [15:14:51] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:00] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:00] RECOVERY - puppet last run on ganeti2004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [15:15:00] RECOVERY - puppet last run on mw2164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:09] RECOVERY - puppet last run on mw2105 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [15:15:09] RECOVERY - puppet last run on elastic2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:10] RECOVERY - puppet last run on rdb2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:19] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [15:15:19] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:15:19] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [15:15:19] RECOVERY - puppet last run on cp3040 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:15:19] RECOVERY - puppet last run on analytics1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:20] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [15:15:20] RECOVERY - puppet last run on db2059 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [15:15:20] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:20] RECOVERY - puppet last run on mw2114 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:15:21] RECOVERY - puppet last run on auth1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:21] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:15:29] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [15:15:29] RECOVERY - puppet last run on ganeti1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:29] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:29] RECOVERY - puppet last run on mw2109 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [15:15:30] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:31] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:15:39] RECOVERY - puppet last run on mw2199 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:40] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:40] RECOVERY - puppet last run on mw2064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:40] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:15:49] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:49] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:49] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:49] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:15:50] RECOVERY - puppet last run on mw2100 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:50] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [15:15:50] RECOVERY - puppet last run on mw2117 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:15:50] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [15:15:51] RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:15:59] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:16:00] RECOVERY - puppet last run on cp3047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:00] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:00] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:01] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:10] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:19] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:20] RECOVERY - puppet last run on mw2083 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:16:20] RECOVERY - puppet last run on db2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:21] RECOVERY - puppet last run on mw2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:21] RECOVERY - puppet last run on wtp2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:21] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:29] RECOVERY - puppet last run on mw2080 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:30] RECOVERY - puppet last run on mw2019 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:16:39] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:40] RECOVERY - puppet last run on mw2082 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:41] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:16:50] RECOVERY - puppet last run on mw2087 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:17:00] RECOVERY - puppet last run on mw2163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:17:00] RECOVERY - puppet last run on mw2015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:17:09] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:17:40] RECOVERY - puppet last run on mw2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:23:39] 06Operations, 07Puppet, 10DBA: Puppet failure on all hosts with Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::Error: Out of range value for column 'id' at row 1: INSERT INTO `fact_values` (`updated_at`, `host_id`, `created... - https://phabricator.wikimedia.org/T107753#2254913 [15:24:41] !log alter table puppet.fact_values to a bigint unsigned for m1 T107753 [15:24:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:30:00] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [15:31:29] PROBLEM - puppet last run on mw2052 is CRITICAL: CRITICAL: puppet fail [15:31:31] PROBLEM - puppet last run on pc1006 is CRITICAL: CRITICAL: puppet fail [15:32:39] jynus: those probably failed while running the alter table: Deadlock found when trying to get lock [15:33:03] I'm forcing a re-run on mw2052 to confirm [15:35:08] yes, thank you [15:35:34] strange because I used online, which means only on trigger creation they are affected [15:35:51] ah, if they are long-running queries they could fail into the metadata locking issue [15:37:53] still running, it's applying configuration version... [15:42:40] I wonder what does it need the id column for [15:43:11] RECOVERY - puppet last run on mw2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:45:09] it finished, took quite a while ^^ [15:57:09] RECOVERY - puppet last run on pc1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:57:40] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:08:38] !log changing live configuration of db1042 thread_pool_stall_limit to 10 to test impact on connection timout [16:08:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:18:28] !log changing live configuration of db1042 thread_pool_stall_limit back to 100 to test impact on connection timeout [16:18:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:20:13] !log changing live configuration of db1042 thread_pool_stall_limit to 10 to avoid connection timeout errors [16:20:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:27:15] you can love the load distribution of the video scalers: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Video%2520scalers%2520eqiad&tab=m&vn=&hide-hf=false [16:28:28] yeah! it all started this morning [16:28:40] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 624 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5372342 keys - replication_delay is 624 [16:30:50] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5331232 keys - replication_delay is 0 [17:03:37] 06Operations, 07Puppet, 10DBA: Puppet failure on all hosts with Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::Error: Out of range value for column 'id' at row 1: INSERT INTO `fact_values` (`updated_at`, `host_id`, `created... - https://phabricator.wikimedia.org/T107753#2254999 [17:11:05] brion, if purge does not reencode, the it is unrelated [17:11:47] but it is too much coincidence that that, commons master issues, and scalers started at the same time [17:13:01] Jynus: we recently added two new (lowest resolution) transcode targets. If someone is running a bunch of purges to uncover the links to reset the new transcodes and resetting the transcodes that might be increasing the load. [17:13:11] I'm not running any such bot at this time [17:13:26] And they do not get scheduled automatically yet [17:13:31] so my thesis would point to something common, like a batch upload that stresses both purges and scalers [17:14:11] I may be talking silly, but I certainly did not investigate [17:14:20] (need a rest now) [17:14:21] :) [17:14:28] Entirely possible yes. [17:14:50] I sent a mail to ops in any case [17:14:54] Do we have logging of active jobs parameters? [17:15:10] yes I think [17:15:25] mediawiki logs should have them [17:15:51] and there are some broad counters on graphite/grafana [17:16:39] Ok we also have what tmh's transcode table thinks is active at https://commons.wikimedia.org/wiki/Special:TimedMediaHandler [17:16:53] Does look like someone's doing some batch uploads of archival footage [17:17:14] Some newer bits too [17:17:36] Shouldn't be too scary but we'll have to learn better how to tune this service [17:17:44] https://grafana.wikimedia.org/dashboard/db/job-queue-rate?panelId=8&fullscreen [17:18:23] Ouch some of these are 2 hour talks [17:18:32] there we have it [17:18:35] That explains why the set is backed up [17:19:25] it is ok, because in reality, it only exposes bugs [17:19:59] :) [17:20:36] but I do not want to do anything something drastic for now, unless platform stability gets compromised [17:20:39] A backup in jobs shouldn't result in higher load than full capacity running [17:20:48] but please share your findings on the ticket [17:20:54] So if we're actually worried about load avg we may need to tune number of threads [17:21:04] I really need to go off :-) [17:21:13] Ok :) [17:21:16] Thanks! [17:41:48] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 622 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5342665 keys - replication_delay is 622 [17:52:09] PROBLEM - puppet last run on mw2067 is CRITICAL: CRITICAL: Puppet has 1 failures [17:52:29] PROBLEM - aqs endpoints health on aqs1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:54:08] PROBLEM - aqs endpoints health on aqs1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:54:20] RECOVERY - aqs endpoints health on aqs1001 is OK: All endpoints are healthy [17:56:18] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] [17:56:48] PROBLEM - aqs endpoints health on aqs1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:02:57] PROBLEM - aqs endpoints health on aqs1001 is CRITICAL: /pageviews/per-article/{project}/{access}/{agent}/{article}/{granularity}/{start}/{end} (Get per article page views) is CRITICAL: Test Get per article page views returned the unexpected status 500 (expecting: 200) [18:03:58] RECOVERY - aqs endpoints health on aqs1001 is OK: All endpoints are healthy [18:05:17] RECOVERY - aqs endpoints health on aqs1003 is OK: All endpoints are healthy [18:05:38] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5336584 keys - replication_delay is 0 [18:06:47] RECOVERY - aqs endpoints health on aqs1002 is OK: All endpoints are healthy [18:16:47] RECOVERY - puppet last run on mw2067 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:17:27] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [19:32:23] 06Operations, 06Discovery, 10Wikidata, 10Wikidata-Query-Service, and 2 others: Reinstall and data reload of WDQS servers - https://phabricator.wikimedia.org/T133566#2255238 (10Smalyshev) [19:32:30] 06Operations, 10ops-eqiad, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: install two Intel 320 Series SSDSA2CW300G3 2.5" 300GB each in wdqs1001/wdqs1002 - https://phabricator.wikimedia.org/T120712#2255241 (10Smalyshev) [19:34:18] PROBLEM - WDQS SPARQL on wdqs1001 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 416 bytes in 0.043 second response time [19:35:08] PROBLEM - WDQS HTTP on wdqs1001 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 416 bytes in 0.006 second response time [19:37:56] !log enabled wdqs1002, put wdqs1001 in maintenance mode for reload [19:38:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:15:22] PROBLEM - Check size of conntrack table on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:15:34] PROBLEM - Disk space on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:15:42] PROBLEM - DPKG on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:15:53] PROBLEM - Labs LDAP on serpens is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:13] PROBLEM - dhclient process on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:16:33] PROBLEM - puppet last run on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:16:52] PROBLEM - salt-minion processes on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:16:53] PROBLEM - RAID on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:17:03] PROBLEM - configured eth on serpens is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:41:01] paravoid: don't wanna give back your op status after 2 weeks opped without spamming from stashbot? [20:41:20] huh? [20:49:43] PROBLEM - SSH on serpens is CRITICAL: Server answer [20:51:42] RECOVERY - SSH on serpens is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [21:16:49] paravoid, +o in the channel [21:16:57] freenode discourages people from staying +o [21:17:47] never heard of that rule, but I avoid it nevertheless [21:19:41] https://freenode.net/changuide [21:22:10] PROBLEM - SSH on serpens is CRITICAL: Server answer [21:29:41] PROBLEM - MariaDB Slave Lag: s1 on db2070 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 312.53 seconds [21:30:20] PROBLEM - MariaDB Slave Lag: s1 on db2042 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 350.53 seconds [21:30:21] PROBLEM - MariaDB Slave Lag: s1 on db2055 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 353.36 seconds [21:30:31] PROBLEM - MariaDB Slave Lag: s1 on db2069 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 364.77 seconds [21:30:39] PROBLEM - MariaDB Slave Lag: s1 on db1053 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 366.24 seconds [21:30:51] PROBLEM - MariaDB Slave Lag: s1 on db2062 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 385.59 seconds [21:31:01] PROBLEM - MariaDB Slave Lag: s1 on db2034 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 397.05 seconds [21:31:20] PROBLEM - MariaDB Slave Lag: s1 on db2048 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 412.53 seconds [21:31:45] PROBLEM - MariaDB Slave Lag: s1 on db2016 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 423.87 seconds [21:31:54] * volans looking [21:31:58] ^^^ [21:33:04] RECOVERY - SSH on serpens is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [21:34:25] RECOVERY - MariaDB Slave Lag: s1 on db2016 is OK: OK slave_sql_lag Replication lag: 0.37 seconds [21:34:55] PROBLEM - MariaDB Slave Lag: s1 on dbstore2002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 627.17 seconds [21:35:44] RECOVERY - MariaDB Slave Lag: s1 on db2055 is OK: OK slave_sql_lag Replication lag: 0.44 seconds [21:35:54] RECOVERY - MariaDB Slave Lag: s1 on db2069 is OK: OK slave_sql_lag Replication lag: 0.41 seconds [21:35:55] RECOVERY - MariaDB Slave Lag: s1 on db2048 is OK: OK slave_sql_lag Replication lag: 0.45 seconds [21:36:24] RECOVERY - MariaDB Slave Lag: s1 on db2070 is OK: OK slave_sql_lag Replication lag: 0.24 seconds [21:36:45] RECOVERY - MariaDB Slave Lag: s1 on dbstore2002 is OK: OK slave_sql_lag Replication lag: 0.23 seconds [21:36:45] RECOVERY - MariaDB Slave Lag: s1 on db2062 is OK: OK slave_sql_lag Replication lag: 0.26 seconds [21:37:14] RECOVERY - MariaDB Slave Lag: s1 on db2042 is OK: OK slave_sql_lag Replication lag: 0.06 seconds [21:37:15] RECOVERY - MariaDB Slave Lag: s1 on db2034 is OK: OK slave_sql_lag Replication lag: 0.43 seconds [21:38:52] RECOVERY - MariaDB Slave Lag: s1 on db1053 is OK: OK slave_sql_lag Replication lag: 30.10 seconds [21:42:15] PROBLEM - NTP on serpens is CRITICAL: NTP CRITICAL: No response from NTP server [21:44:34] PROBLEM - SSH on serpens is CRITICAL: Server answer [22:03:17] RECOVERY - SSH on serpens is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [22:18:21] stashbot: can you tell me about T134098 ? [22:21:32] stashbot: how about now? T134098 [22:21:33] T134098: Furud has flapping most of the day - https://phabricator.wikimedia.org/T134098 [22:26:17] PROBLEM - SSH on serpens is CRITICAL: Server answer [22:53:01] RECOVERY - SSH on serpens is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [23:00:41] PROBLEM - SSH on serpens is CRITICAL: Server answer [23:19:36] RECOVERY - SSH on serpens is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0) [23:32:30] PROBLEM - SSH on serpens is CRITICAL: Server answer [23:48:10] PROBLEM - puppet last run on mw2112 is CRITICAL: CRITICAL: Puppet has 1 failures