[00:02:58] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:06] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:07:56 UTC 2013 [00:08:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:50] !log dist-upgrading nitrogen [00:08:58] Logged the message, Master [00:09:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:09:07 UTC 2013 [00:09:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:46] PROBLEM - Host nitrogen is DOWN: CRITICAL - Host Unreachable (208.80.154.17) [00:10:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:10:10 UTC 2013 [00:10:16] RECOVERY - Host nitrogen is UP: PING OK - Packet loss = 0%, RTA = 1.14 ms [00:10:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:11:11 UTC 2013 [00:11:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:06] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:12:01 UTC 2013 [00:12:28] !log dist-upgrading yvon [00:12:36] Logged the message, Master [00:12:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:56] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:12:45 UTC 2013 [00:13:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:46] PROBLEM - Apache HTTP on mw1058 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:56] PROBLEM - Host yvon is DOWN: PING CRITICAL - Packet loss = 100% [00:14:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.121 second response time [00:14:56] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 00:14:46 UTC 2013 [00:15:16] RECOVERY - Host yvon is UP: PING OK - Packet loss = 0%, RTA = 26.72 ms [00:15:46] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:07] New patchset: Odder; "(bug 47749) Categorise {{#babel}} on udmwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64467 [00:56:18] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:28] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.0005874633789 secs [01:02:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:28] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset 0.0007549524307 secs [01:03:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [01:22:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:37] !log singer boot issue - affects service contacts.wm (but no others) [01:22:46] Logged the message, Master [01:23:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [01:31:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [01:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [01:59:23] New patchset: Asher; "add binlog control options to mysql_multi_instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64470 [02:03:37] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64470 [02:06:30] !log LocalisationUpdate completed (1.22wmf4) at Sat May 18 02:06:30 UTC 2013 [02:06:39] Logged the message, Master [02:11:28] !log LocalisationUpdate completed (1.22wmf3) at Sat May 18 02:11:28 UTC 2013 [02:11:38] Logged the message, Master [02:25:29] RECOVERY - mysqld processes on db1053 is OK: PROCS OK: 1 process with command name mysqld [02:28:19] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [02:29:09] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat May 18 02:29:09 UTC 2013 [02:29:17] Logged the message, Master [02:36:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:37:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [02:52:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [03:13:00] Bugzilla feels slow right now. [03:16:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [03:26:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.147 second response time [03:40:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:41:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [03:57:35] New patchset: Asher; "disable linux native aio for multi instances, use unique tmp dirs, fix client socket stanza" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64471 [03:58:45] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64471 [04:01:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [04:02:57] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [04:02:57] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [04:02:57] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:05] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:07:57 UTC 2013 [04:08:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:15] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:09:07 UTC 2013 [04:09:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:38] Coren|Sleep: encountered some issues with async io with the new multi-mysql instance setup but that's resolved and sanitized versions of enwiki / dewki / wikidata are currently copying over to the actual labsdb hosts [04:10:15] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:10:09 UTC 2013 [04:10:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:15] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:11:05 UTC 2013 [04:11:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:05] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:11:56 UTC 2013 [04:12:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:45] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:12:39 UTC 2013 [04:13:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:55] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 04:14:47 UTC 2013 [04:15:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:17:05] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [05:25:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [05:31:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:32:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [05:40:34] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:41:34] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [06:14:30] RECOVERY - mysqld processes on labsdb1001 is OK: PROCS OK: 1 process with command name mysqld [06:29:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 06:28:57 UTC 2013 [06:29:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:29:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 06:29:45 UTC 2013 [06:30:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 06:30:56 UTC 2013 [06:31:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [06:45:49] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [07:01:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [07:31:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.171 second response time [07:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [08:01:19] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.001273036003 secs [08:07:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 08:07:46 UTC 2013 [08:08:22] RECOVERY - Puppet freshness on mc15 is OK: puppet ran at Sat May 18 08:08:13 UTC 2013 [08:08:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:32] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 08:08:27 UTC 2013 [08:09:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:22] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:22] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 08:14:51 UTC 2013 [08:15:22] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:32] New review: Nikerabbit; "Just a note that FR is quite ambiguous in this context." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64347 [08:32:52] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.002845644951 secs [08:40:01] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [10:03:02] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:00] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:03] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:07:55 UTC 2013 [12:08:24] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:13] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:09:04 UTC 2013 [12:09:23] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:13] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:10:07 UTC 2013 [12:10:24] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:03] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:11:02 UTC 2013 [12:11:23] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:53] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:11:51 UTC 2013 [12:12:23] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:43] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:12:33 UTC 2013 [12:13:24] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:53] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 12:14:45 UTC 2013 [12:15:23] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:29:03] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [14:03:01] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [14:03:01] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [14:03:01] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [14:17:57] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [14:38:22] PROBLEM - SSH on gadolinium is CRITICAL: Server answer: [14:38:42] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [14:39:12] PROBLEM - SSH on cp1044 is CRITICAL: Server answer: [14:39:23] PROBLEM - SSH on cp1043 is CRITICAL: Server answer: [14:40:22] RECOVERY - SSH on cp1043 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:42:12] RECOVERY - SSH on cp1044 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:43:22] RECOVERY - SSH on gadolinium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:49:42] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [16:08:06] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:07:59 UTC 2013 [16:08:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:09:09 UTC 2013 [16:09:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:10:12 UTC 2013 [16:10:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:11:10 UTC 2013 [16:11:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:06] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:12:00 UTC 2013 [16:12:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:46] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:12:43 UTC 2013 [16:13:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:56] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 16:14:47 UTC 2013 [16:15:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:47] New patchset: Catrope; "Clean up VisualEditor config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64493 [16:57:10] New review: Catrope; "Do not merge this yet, needs Parsoid config changes" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/64493 [17:09:54] New patchset: Asher; "labsdbs: disable binlogs, relax locking, bufferpool sized for hw" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64495 [17:10:46] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64495 [17:16:18] New patchset: Asher; "log_bin should default to false" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64496 [17:16:44] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64496 [18:14:23] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [18:14:23] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [18:34:13] New patchset: Odder; "(bug 47574) Change namespace settings for cewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64501 [18:40:11] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [19:37:11] New patchset: QChris; "Stop gerrit's commit link detection from mangling MediaWiki Urls" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64502 [19:41:35] New review: Matmarex; "(1 comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/64502 [19:44:22] New review: MZMcBride; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64502 [19:45:00] Oh, it was intentional. [19:45:01] hurr [19:46:56] New review: MZMcBride; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64502 [20:03:50] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:07:54 UTC 2013 [20:08:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:09:02 UTC 2013 [20:09:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:10:03 UTC 2013 [20:10:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:11:03 UTC 2013 [20:11:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:11:53 UTC 2013 [20:12:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:19] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:13:11 UTC 2013 [20:13:39] PROBLEM - Redis on mc15 is CRITICAL: Connection timed out [20:13:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:30] RECOVERY - Redis on mc15 is OK: TCP OK - 0.027 second response time on port 6379 [20:14:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 18 20:14:45 UTC 2013 [20:14:59] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:57:07] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:00] New patchset: QChris; "Stop gerrit's commit link detection from mangling MediaWiki Urls" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64502 [21:12:01] wee qchris [21:12:26] :-) [21:13:04] But that is more of a workaround than a real fix. RegExp are gruesome in gerrit :-( [22:27:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [22:29:56] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [23:00:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:02:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [23:17:29] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [23:41:05] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [23:54:55] PROBLEM - SSH on stat1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:45] RECOVERY - SSH on stat1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)