[00:04:48] RobH: :-( [00:04:53] I miss them both. [00:05:29] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [00:05:39] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.158290863 secs [00:32:39] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 00:32:36 UTC 2013 [00:33:29] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [00:34:39] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [00:35:52] (03CR) 10Reedy: [C: 031] Rebuild localisation cache in several threads [operations/puppet] - 10https://gerrit.wikimedia.org/r/79231 (owner: 10MaxSem) [01:04:21] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [01:09:11] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:41] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1311465502 secs [01:41:09] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [01:57:39] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [02:06:07] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [02:14:54] !log LocalisationUpdate completed (1.22wmf12) at Sat Aug 17 02:14:54 UTC 2013 [02:15:07] Logged the message, Master [02:15:57] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:57] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:57] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:57] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:57] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:58] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [02:19:59] !log LocalisationUpdate completed (1.22wmf13) at Sat Aug 17 02:19:58 UTC 2013 [02:20:10] Logged the message, Master [02:32:47] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 02:32:42 UTC 2013 [02:33:07] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [02:34:37] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1442755461 secs [02:40:08] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 17 02:40:08 UTC 2013 [02:40:20] Logged the message, Master [03:03:40] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [03:06:12] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [03:32:43] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 03:32:41 UTC 2013 [03:33:12] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [04:00:38] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1349291801 secs [04:05:30] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [04:32:50] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 04:32:40 UTC 2013 [04:33:30] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [05:06:24] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [05:14:10] (03CR) 10MZMcBride: ":-(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/79443 (owner: 10RobH) [05:32:44] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 05:32:39 UTC 2013 [05:33:24] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [05:51:15] PROBLEM - Host mw72 is DOWN: PING CRITICAL - Packet loss = 100% [05:51:55] RECOVERY - Host mw72 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [06:05:54] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [06:10:04] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:04] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [06:32:44] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 06:32:38 UTC 2013 [06:32:54] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [07:06:28] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [07:12:59] paravoid: hi, if you have some time please have a look at https://gerrit.wikimedia.org/r/#/c/79453/ [07:13:12] paravoid: we're trying to add a dtree for backwards compatibility and make new packages [07:33:18] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 07:33:14 UTC 2013 [07:33:28] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [07:55:45] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [07:58:15] RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 27.10 ms [08:00:35] PROBLEM - search indices - check lucene status page on search23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:25] RECOVERY - search indices - check lucene status page on search23 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.055 second response time [08:05:51] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [08:13:11] PROBLEM - NTP on search23 is CRITICAL: NTP CRITICAL: Offset unknown [08:17:11] RECOVERY - NTP on search23 is OK: NTP OK: Offset -0.0003650188446 secs [08:22:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [08:32:41] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 08:32:35 UTC 2013 [08:32:51] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [08:44:05] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [08:51:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:01:45] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [09:06:42] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [09:09:42] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1901636124 secs [09:21:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [09:32:45] (03PS1) 10TTO: Enable "block" action for AbuseFilter on meta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/79458 [09:33:02] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 09:32:52 UTC 2013 [09:33:42] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [09:52:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [10:00:21] (03CR) 10Nikerabbit: [C: 031] Remove long-buried $wgLogAutocreatedAccounts [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/79059 (owner: 10Nemo bis) [10:06:59] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [10:15:21] (03PS1) 10Akosiaris: bacula module fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/79461 [10:19:33] (03CR) 10Akosiaris: [C: 032] bacula module fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/79461 (owner: 10Akosiaris) [10:21:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [10:25:09] RECOVERY - Puppet freshness on helium is OK: puppet ran at Sat Aug 17 10:25:00 UTC 2013 [10:32:49] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 10:32:42 UTC 2013 [10:32:59] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [10:40:40] (03PS1) 10Akosiaris: Change bacula SD paths [operations/puppet] - 10https://gerrit.wikimedia.org/r/79463 [10:43:11] (03CR) 10Akosiaris: [C: 032] Change bacula SD paths [operations/puppet] - 10https://gerrit.wikimedia.org/r/79463 (owner: 10Akosiaris) [10:51:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [10:57:38] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [11:01:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [11:03:35] (03PS1) 10Akosiaris: Adding a forgotten -1st- stanza [operations/puppet] - 10https://gerrit.wikimedia.org/r/79464 [11:04:23] (03CR) 10Akosiaris: [C: 032] Adding a forgotten -1st- stanza [operations/puppet] - 10https://gerrit.wikimedia.org/r/79464 (owner: 10Akosiaris) [11:04:38] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1222013235 secs [11:05:34] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [11:09:14] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [11:31:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [11:32:44] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 11:32:41 UTC 2013 [11:33:34] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [11:51:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [12:05:09] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [12:15:59] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:59] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:59] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:59] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:59] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:00] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [12:21:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [12:32:39] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 12:32:35 UTC 2013 [12:33:09] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [12:36:34] (03PS1) 10Raimond Spekking: Update domain for Elya&Raymond blog [operations/puppet] - 10https://gerrit.wikimedia.org/r/79466 [12:45:37] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [12:52:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:55:37] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.2054158449 secs [13:05:19] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [13:25:19] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:09] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [13:32:49] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 13:32:39 UTC 2013 [13:33:19] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [13:45:38] paravoid: Thank you for writing up that detailed e-mail to wikitech-l. :-) [13:52:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [13:57:39] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [14:04:39] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.128999114 secs [14:06:04] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [14:21:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [14:32:54] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 14:32:45 UTC 2013 [14:33:04] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [14:51:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [14:58:36] (03PS1) 10Springle: return db1009 to the pool after resync [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/79470 [14:59:41] (03CR) 10Springle: [C: 032 V: 032] return db1009 to the pool after resync [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/79470 (owner: 10Springle) [15:02:12] !log springle synchronized wmf-config/db-eqiad.php 'returning db1009 to the pool after sync' [15:02:24] Logged the message, Master [15:06:08] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [15:21:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [15:32:49] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 15:32:42 UTC 2013 [15:33:08] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [15:38:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [16:06:27] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [16:10:27] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:27] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [16:13:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [16:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [16:32:47] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 16:32:40 UTC 2013 [16:33:27] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [16:40:23] Elsie: yvw [16:43:25] PROBLEM - SSH on pdf1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:16] RECOVERY - SSH on pdf1 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [16:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [17:05:31] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [17:06:31] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [17:09:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [17:13:32] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1143738031 secs [17:21:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [17:32:41] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 17:32:40 UTC 2013 [17:33:32] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [17:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [18:05:42] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [18:24:33] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [18:29:55] apergos, I hope all db1009 tables were checked and not only nlwiki one [18:30:05] yes [18:30:14] all dbs and all tables [18:30:33] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1092483997 secs [18:30:45] those db corruption errors are ugly [18:30:48] the hiccup was too big to risk anything else [18:30:51] yes indeed [18:31:48] actually we were lucky that an high profile page was affected and thus noticed by users [18:32:31] I had seen toolserver replication 'losing' statements but not on wikipedia [18:32:52] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 18:32:43 UTC 2013 [18:33:42] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [18:33:42] and yeah it shouldn't have differed and yet guess what, it did [18:33:46] I saw it with my own eyes [18:33:50] so ... [18:33:57] (about page_random) [18:34:10] for a few hundred pages [18:34:12] no good [18:36:16] and both servers were the same architecture [18:36:26] I don't think we have any more 32/64 bit or similar [18:36:59] perhaps the fpus have different precision [18:38:01] it's a slave server, it should just have the identical data [18:38:05] as all the other slaves did have [18:38:14] this was db corruption pure and simple [18:38:47] I would like to know how it wound up like that but I would probably lose my sanity trying to figure it out [18:39:31] I suspect we will see it again [18:39:44] hope not [18:40:19] we don't see those sort of lockups very often and it's a good thing too, next time we'll likly not even bother to examine all the tables but just sync them up right away (I wuld imagine) [18:45:05] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [19:06:01] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [19:32:41] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 19:32:35 UTC 2013 [19:33:01] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [20:08:52] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [20:32:52] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 20:32:42 UTC 2013 [20:32:52] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [20:47:33] ori-l: i think not. you could do general query log to disk (or maybe to somewhere else?) but would need root to enable it. or you could tcpdump and parse that into query log (percona toolkit has the parser for that) but that also requires root [20:47:43] oh, wow, i'm behind [20:48:35] it's possible to use https://dev.mysql.com/doc/refman/5.0/en/mysqlbinlog.html [20:55:43] ori-l: oh, i thought maybe you wanted reads too [20:56:07] nope [20:56:08] as you'd see in show processlist [20:56:11] k [20:56:16] then binglog is a good option [20:56:27] and you could even get it over the network with the right password [21:05:34] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [21:09:24] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [21:09:35] PROBLEM - NTP on pdf3 is CRITICAL: NTP CRITICAL: Offset unknown [21:17:34] RECOVERY - NTP on pdf3 is OK: NTP OK: Offset 0.1370099783 secs [21:32:45] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 21:32:42 UTC 2013 [21:33:34] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [21:58:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [22:10:48] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [22:16:28] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [22:16:28] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [22:16:28] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [22:16:28] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [22:16:28] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [22:16:29] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [22:32:48] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 22:32:42 UTC 2013 [22:33:48] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [23:05:59] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [23:33:19] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sat Aug 17 23:33:15 UTC 2013 [23:34:00] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours