[00:10:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:12:57] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [00:13:51] * AaronSchulz wonders why his rgw broke [00:15:58] nvm [00:21:57] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [00:21:57] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [00:21:57] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [00:21:57] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [00:23:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.596 seconds [00:31:18] preilly: here [00:31:23] rfaulkner: welcome to the real channel [00:31:31] haha ty [00:31:31] rfaulkner: so this is the one that you can use https://github.com/wikimedia/Sartoris [00:32:36] PROBLEM - Host wikipedia-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::1 [00:32:37] PROBLEM - Host wikiquote-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::3 [00:32:38] PROBLEM - Host mediawiki-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::8 [00:32:45] PROBLEM - Swift HTTP on ms-fe1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:57] hrm [00:33:30] PROBLEM - Host wikiversity-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::7 [00:33:48] PROBLEM - Host upload-lb.esams.wikimedia.org_ipv6_https is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::b [00:33:49] PROBLEM - Host wikiversity-lb.esams.wikimedia.org_ipv6_https is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::7 [00:33:54] fuck, it looks like cr1-eqiad's fpc rebooted [00:33:55] again [00:33:55] Ryan_Lane: every time I hear "swift" my ears twitch [00:33:57] PROBLEM - Host bits-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::a [00:34:15] PROBLEM - Host wikisource-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::5 [00:34:15] PROBLEM - Host wiktionary-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::2 [00:34:16] PROBLEM - Host foundation-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::9 [00:34:16] PROBLEM - Host wikinews-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::6 [00:34:24] PROBLEM - Swift HTTP on ms-fe1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:33] AaronSchulz: :D [00:34:33] PROBLEM - Host upload-lb.esams.wikimedia.org_ipv6 is DOWN: /bin/ping6 -n -U -w 15 -c 5 2620:0:862:ed1a::b [00:34:51] RECOVERY - Host bits-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 115.55 ms [00:34:51] RECOVERY - Host foundation-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 116.90 ms [00:35:00] RECOVERY - Host upload-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 117.52 ms [00:35:01] RECOVERY - Host wikisource-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 115.58 ms [00:35:09] RECOVERY - Host wikinews-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 116.84 ms [00:35:11] RECOVERY - Host wiktionary-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 117.32 ms [00:35:11] RECOVERY - Host wikiversity-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 115.97 ms [00:35:28] RECOVERY - Host wikiquote-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 115.59 ms [00:35:29] RECOVERY - Host wikipedia-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 115.91 ms [00:35:36] RECOVERY - Host mediawiki-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 117.37 ms [00:36:48] !log appears that cr1-eqiad's fpc rebooted [00:36:56] Logged the message, Mistress of the network gear. [00:37:08] yeah it's not the first time [00:37:15] I was debugging it the other day too [00:37:30] then figured there's nothing I can do and told ma rk, not sure if he told you [00:39:30] RECOVERY - Host upload-lb.esams.wikimedia.org_ipv6_https is UP: PING OK - Packet loss = 0%, RTA = 114.36 ms [00:39:31] RECOVERY - Host wikiversity-lb.esams.wikimedia.org_ipv6_https is UP: PING OK - Packet loss = 0%, RTA = 112.42 ms [00:40:01] yeah [00:40:05] :( [00:41:45] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 185 seconds [00:42:48] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 187 seconds [00:56:00] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 189 seconds [00:56:27] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 196 seconds [00:57:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:09] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 17 seconds [01:12:03] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [01:12:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.719 seconds [01:21:31] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 262 seconds [01:23:09] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [01:24:03] PROBLEM - Swift HTTP on ms-fe1004 is CRITICAL: HTTP CRITICAL - No data received from host [02:25:26] !log LocalisationUpdate completed (1.21wmf6) at Sat Dec 15 02:25:26 UTC 2012 [02:25:37] Logged the message, Master [02:27:17] so if juniper doesn't get back to me soon, i'll disable bgp and switch ospf costs on cr1 to make all traffic go to cr2-eqiad [02:27:22] it'll fuck our commits though :( [02:35:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:03] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.930 seconds [02:41:25] New review: Catrope; "DNS for this is done now. Note that when deploying this change, a Pybal restart (per https://wikitec..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/38457 [02:45:56] !log drained all transit traffic from cr1-eqiad -- restore by load override /var/home/lcarr/cr1-eqiad.undrained if traffic levels seem too high [02:46:05] Logged the message, Mistress of the network gear. [02:47:17] !log LocalisationUpdate completed (1.21wmf5) at Sat Dec 15 02:47:17 UTC 2012 [02:47:26] Logged the message, Master [03:32:45] RECOVERY - Puppet freshness on erzurumi is OK: puppet ran at Sat Dec 15 03:32:38 UTC 2012 [03:57:48] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [06:00:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.998 seconds [06:06:08] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [06:14:05] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [06:37:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.806 seconds [06:51:26] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [07:20:05] PROBLEM - Swift HTTP on ms-fe1003 is CRITICAL: HTTP CRITICAL - No data received from host [07:23:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.051 seconds [08:10:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:25:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [08:57:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.045 seconds [09:44:32] PROBLEM - Host ms-be1 is DOWN: PING CRITICAL - Packet loss = 100% [09:46:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:21] New review: Hashar; "recheck" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/38797 [09:55:28] New review: Hashar; "recheck" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/24620 [09:59:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.595 seconds [10:13:47] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [10:15:44] PROBLEM - Swift HTTP on ms-fe1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:47] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [10:22:47] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [10:22:48] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [10:22:48] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [10:33:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.687 seconds [11:21:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.044 seconds [11:39:40] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:39:40] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [11:55:43] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [11:55:44] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [12:08:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:49] !log ms-be1 can't be reached form mgmt console to check on it (Connection refused) [12:19:02] Logged the message, Master [12:25:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [12:57:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.869 seconds [13:24:10] PROBLEM - Swift HTTP on ms-fe1004 is CRITICAL: HTTP CRITICAL - No data received from host [13:35:59] apergos: mmm [13:44:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:39] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [14:00:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [14:20:06] PROBLEM - Swift HTTP on ms-fe1003 is CRITICAL: HTTP CRITICAL - No data received from host [14:30:22] !log powercycling ms-be1 [14:30:31] Logged the message, Master [14:33:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:18] RECOVERY - Host ms-be1 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [14:37:18] !log maxsem synchronized /php-1.21wmf5/includes/Message.php 'https://gerrit.wikimedia.org/r/#/c/38822/' [14:37:26] Logged the message, Master [14:49:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.042 seconds [15:22:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.038 seconds [16:07:39] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [16:10:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:39] PROBLEM - check_gcsip on payments1 is CRITICAL: Connection timed out [16:10:39] PROBLEM - check_gcsip on payments4 is CRITICAL: Connection timed out [16:10:40] PROBLEM - check_gcsip on payments3 is CRITICAL: Connection timed out [16:10:40] PROBLEM - check_gcsip on payments2 is CRITICAL: Connection timed out [16:15:27] RECOVERY - check_gcsip on payments2 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.139 second response time [16:15:27] RECOVERY - check_gcsip on payments3 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.157 second response time [16:15:28] RECOVERY - check_gcsip on payments4 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.192 second response time [16:15:28] RECOVERY - check_gcsip on payments1 is OK: HTTP OK: HTTP/1.1 200 OK - 378 bytes in 0.165 second response time [16:15:36] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [16:22:48] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 209 seconds [16:23:24] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 228 seconds [16:26:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.183 seconds [16:36:18] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [16:37:12] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [16:45:36] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [16:52:39] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [16:58:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds [17:20:06] PROBLEM - Swift HTTP on ms-fe1003 is CRITICAL: HTTP CRITICAL - No data received from host [17:36:44] New review: MZMcBride; "This is a trivial changeset. What needs to happen to get this deployed? Is there an associated RT ti..." [operations/debs/squid] (master) C: 0; - https://gerrit.wikimedia.org/r/18331 [17:47:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [18:08:15] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 218 seconds [18:08:33] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 229 seconds [18:28:37] New review: Pgehres; "I'm not in a terrible hurry as this isn't really the best solution. Since most error pages occur wh..." [operations/debs/squid] (master) C: 0; - https://gerrit.wikimedia.org/r/18331 [18:36:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.041 seconds [19:19:30] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [19:20:15] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [19:24:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.575 seconds [20:13:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:14:33] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Puppet has not run in the last 10 hours [20:23:33] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [20:23:33] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [20:23:34] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [20:23:34] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [20:26:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.797 seconds [20:33:15] paravoid: I could not get to mgmt either from fenari or bast1001 (and obviously not to the box either) [20:33:33] thanks for rebooting [20:55:37] apergos: because you tried SSH on a C2100 [20:55:56] ah ;-D [20:56:10] forgot! so used to having the 720s already [20:56:42] that I've wiped the weirdness of the c2100s (but not their brokenness) out of my mind already... [21:00:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:47] WTF is going on with stafford? probably half of Nagios spam is from it [21:16:48] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [21:40:39] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:40:40] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [21:50:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:33] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [21:56:33] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [22:03:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.737 seconds [22:22:12] PROBLEM - Swift HTTP on ms-fe1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:30] PROBLEM - Swift HTTP on ms-fe1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:12] PROBLEM - MySQL Replication Heartbeat on db26 is CRITICAL: CRIT replication delay 207 seconds [22:46:57] PROBLEM - MySQL Slave Delay on db26 is CRITICAL: CRIT replication delay 238 seconds [22:49:30] RECOVERY - MySQL Replication Heartbeat on db26 is OK: OK replication delay 0 seconds [22:50:15] RECOVERY - MySQL Slave Delay on db26 is OK: OK replication delay 0 seconds [22:54:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [23:09:36] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [23:26:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:39:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.347 seconds [23:59:33] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours