[00:00:19] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [00:05:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:08:39] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:08:34 UTC 2013 [00:08:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:09:39] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:09:37 UTC 2013 [00:09:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:10:39] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:10:32 UTC 2013 [00:10:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:11:30] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:11:22 UTC 2013 [00:11:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:12:09] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:12:04 UTC 2013 [00:12:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:12:49] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:12:41 UTC 2013 [00:13:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:31:19] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [00:32:59] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 00:32:52 UTC 2013 [00:33:49] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [01:03:04] uhh [01:03:04] ^ these comments were posted 45 minutes ago. [01:03:04] so you guys might want to look into it. [01:03:19] Ryan_Lane: why is there no booze? [01:03:19] because we're not doing drinks in the office today [01:05:40] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [01:55:28] !log iwlinks iwl_prefix_from_title migration completed for all projects [01:55:36] Logged the message, Master [02:04:32] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [02:11:57] !log LocalisationUpdate completed (1.22wmf1) at Sat Apr 6 02:11:56 UTC 2013 [02:12:05] Logged the message, Master [02:13:12] PROBLEM - LVS HTTP IPv4 on appservers.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1600 bytes in 2.141 second response time [02:13:28] English Wikipedia: Cannot contact the database server: Unknown error (10.64.16.6) [02:13:52] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1600 bytes in 2.196 second response time [02:14:02] PROBLEM - Apache HTTP on mw1107 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:02] PROBLEM - Apache HTTP on mw1059 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:02] PROBLEM - MySQL Replication Heartbeat on db1017 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:14:02] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:02] PROBLEM - Apache HTTP on mw1113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:03] PROBLEM - Apache HTTP on mw1074 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:03] PROBLEM - Apache HTTP on mw1161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:04] PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1600 bytes in 2.172 second response time [02:14:12] PROBLEM - LVS HTTP IPv4 on appservers.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:14] PROBLEM - Apache HTTP on mw1184 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:14] PROBLEM - Apache HTTP on mw1210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:14] PROBLEM - Apache HTTP on mw1084 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:14] PROBLEM - MySQL Slave Delay on db1017 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:14:14] PROBLEM - Apache HTTP on mw1037 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:14] AndrewBognott, ^ [02:14:15] PROBLEM - Apache HTTP on mw1083 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:15] PROBLEM - Apache HTTP on mw1215 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:16] PROBLEM - Apache HTTP on mw1106 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:16] PROBLEM - Apache HTTP on mw1080 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:17] PROBLEM - Apache HTTP on mw1067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:17] PROBLEM - Apache HTTP on mw1057 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:18] PROBLEM - Apache HTTP on mw1041 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:22] PROBLEM - Apache HTTP on mw1175 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:22] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:22] PROBLEM - Apache HTTP on mw1182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:22] PROBLEM - Apache HTTP on mw1213 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:22] PROBLEM - Apache HTTP on mw1102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:23] PROBLEM - Apache HTTP on mw1087 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:23] PROBLEM - Apache HTTP on mw1035 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:24] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:24] PROBLEM - Apache HTTP on mw1111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:24] AndrewBogott [02:14:37] Webpage now completely down [02:14:50] Back up [02:15:33] RECOVERY - Apache HTTP on mw1033 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.052 second response time [02:15:33] RECOVERY - Apache HTTP on mw1176 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.053 second response time [02:15:33] RECOVERY - Apache HTTP on mw1021 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [02:15:33] RECOVERY - Apache HTTP on mw1031 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.065 second response time [02:15:33] RECOVERY - Apache HTTP on mw1040 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.062 second response time [02:15:33] RECOVERY - Apache HTTP on mw1047 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [02:15:54] RECOVERY - Apache HTTP on mw1187 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.056 second response time [02:15:54] RECOVERY - Apache HTTP on mw1107 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [02:15:54] RECOVERY - Apache HTTP on mw1090 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.063 second response time [02:15:54] RECOVERY - Apache HTTP on mw1060 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.070 second response time [02:15:54] RECOVERY - Apache HTTP on mw1113 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.072 second response time [02:15:54] RECOVERY - Apache HTTP on mw1161 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [02:16:02] RECOVERY - Apache HTTP on mw1210 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [02:16:02] RECOVERY - Apache HTTP on mw1106 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [02:16:02] RECOVERY - Apache HTTP on mw1067 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.073 second response time [02:22:49] !log LocalisationUpdate completed (1.21wmf12) at Sat Apr 6 02:22:48 UTC 2013 [02:22:56] Logged the message, Master [03:05:04] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [03:28:24] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [03:28:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [03:31:24] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [04:09:19] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:13:49] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [04:16:49] LeslieCarr: looking at the xenon flapping discussion: is icinga/naggen/etc. cloned in labs in a way that's useful to hack on prod? [04:28:20] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [04:31:19] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [04:41:11] PROBLEM - Squid on brewster is CRITICAL: Connection refused [05:05:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [05:08:38] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [05:22:18] RECOVERY - Squid on brewster is OK: TCP OK - 0.026 second response time on port 8080 [05:22:29] from backlog it seems there was a 3ish min outage earlier [05:22:44] watchmouse is green [05:28:18] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [05:31:38] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [05:54:32] rfaulkner: please comment on gerrit instead of sending private mails! :-) [05:55:26] jeremyb_: will do! [05:57:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [06:05:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:27:51] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:21] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 06:30:11 UTC 2013 [06:30:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:31:51] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 06:31:44 UTC 2013 [06:32:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:33:11] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 06:33:08 UTC 2013 [06:33:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:34:31] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 06:34:26 UTC 2013 [06:34:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:35:43] New review: Jeremyb; "Copying from private mail." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56576 [06:46:51] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [07:01:51] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [07:05:31] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [07:36:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:37:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [08:05:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:08:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:08:21 UTC 2013 [08:09:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:09:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:09:43 UTC 2013 [08:10:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:11:08] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:11:04 UTC 2013 [08:11:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:12:19] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:12:13 UTC 2013 [08:13:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:13:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:13:23 UTC 2013 [08:14:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:14:19] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:14:16 UTC 2013 [08:15:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:15:58] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:15:51 UTC 2013 [08:16:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:16:38] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:16:30 UTC 2013 [08:17:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:32:58] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 08:32:48 UTC 2013 [08:33:18] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:48:51] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [09:01:51] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [09:04:51] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [09:06:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [09:31:47] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [09:47:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:47:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:47:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:01:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [10:05:17] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [10:18:07] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [10:31:07] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [10:31:57] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:48:11] New review: Ori.livneh; "(1 comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/56104 [11:06:02] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [11:11:22] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [11:32:22] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [11:44:22] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [12:02:22] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:05:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:08:31] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:08:27 UTC 2013 [12:08:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:09:41] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:09:31 UTC 2013 [12:09:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:10:41] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:10:31 UTC 2013 [12:10:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:11:21] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:11:19 UTC 2013 [12:11:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:12:11] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:12:04 UTC 2013 [12:12:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:13:21] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:13:11 UTC 2013 [12:13:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:15:21] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [12:19:01] <^demon> !log restarting gerrit again, stream-events halted like friday. [12:19:08] Logged the message, Master [12:26:56] <^demon> !log also restarted zuul [12:27:03] Logged the message, Master [12:28:27] <^demon> Ok, now everything seems humming along again. Damn we need to fix that Monday. [12:32:21] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:33:11] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 12:33:04 UTC 2013 [12:33:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:45:22] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [13:05:52] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [13:33:22] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [13:36:23] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:04:11] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [14:10:11] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [14:11:21] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [14:14:01] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [15:05:44] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [15:29:14] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [15:32:38] New review: Odder; "Should I even approve it? :-)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/57497 [16:05:43] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [16:06:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:08:34] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:08:32 UTC 2013 [16:09:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:09:43] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:09:34 UTC 2013 [16:10:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:10:34] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:10:31 UTC 2013 [16:11:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:11:23] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:11:21 UTC 2013 [16:12:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:12:13] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:12:05 UTC 2013 [16:13:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:13:13] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:13:11 UTC 2013 [16:14:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:14:33] mutante-away: [16:20:57] Any ideas what this means: [16:21:04] "Although this PHP script (/w/index.php) exists, the file requested for output (mwstore://local-swift/local-deleted/m/u/l/muledfmoybzjk6kp80vmyg22xeb3j3e.png) does not." [16:28:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:23] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 16:33:16 UTC 2013 [16:34:03] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:35:43] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [17:06:41] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [17:31:41] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [18:06:46] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [18:57:00] New patchset: Ori.livneh; "Puppetize scap-1 and scap-2 scripts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57854 [19:04:43] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [19:08:15] ori-l: it's weekend :) [19:08:45] hey drdee [19:09:37] what's up? [19:11:26] nothing, just saying hello [19:23:03] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:23:58] New patchset: Siebrand; "Update TTMServer Solr schema" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57498 [19:34:03] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [19:47:03] PROBLEM - LVS HTTPS IPv4 on mobile-lb.eqiad.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.018 second response time [19:47:43] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [19:47:43] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [19:47:43] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [19:47:53] RECOVERY - LVS HTTPS IPv4 on mobile-lb.eqiad.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 19721 bytes in 0.009 second response time [19:55:03] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:06:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:08:40] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:08:35 UTC 2013 [20:09:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:09:40] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:09:37 UTC 2013 [20:10:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:10:40] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:10:32 UTC 2013 [20:11:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:11:30] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:11:22 UTC 2013 [20:12:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:12:54] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:12:41 UTC 2013 [20:13:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:13:11] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:13:09 UTC 2013 [20:14:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:32:10] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [20:33:00] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Sat Apr 6 20:32:52 UTC 2013 [20:33:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [20:33:40] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [20:36:48] New patchset: Ori.livneh; "Set common rsync and dsh parameters in mw-deployment-vars" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57890 [20:48:44] New patchset: Ori.livneh; "Puppetize scap-1 and scap-2 scripts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57854 [20:49:06] New patchset: Ori.livneh; "Set common rsync and dsh parameters in mw-deployment-vars" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57890 [20:49:40] New patchset: Ori.livneh; "Set common rsync and dsh parameters in mw-deployment-vars" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57890 [21:03:44] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [21:06:02] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [21:14:42] PROBLEM - Disk space on cp1041 is CRITICAL: Timeout while attempting connection [21:16:42] RECOVERY - Disk space on cp1041 is OK: DISK OK [22:05:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [22:33:29] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [22:36:35] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [23:02:26] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [23:03:51] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [23:04:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:10] jeremyb_: the icinga class is pretty complete - and naggen is in puppetmaster.pp: source => "puppet:///files/puppet/naggen" [23:05:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.818 second response time [23:37:24] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [23:49:03] there seem to be some thumbnail purging issues, see my email to the ops mailing list.