[00:08:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 00:07:53 UTC 2013 [00:08:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 00:08:40 UTC 2013 [00:09:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 00:09:21 UTC 2013 [00:10:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 00:14:46 UTC 2013 [00:15:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:11] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [07:55:05] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:08:07 UTC 2013 [08:08:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:09:28 UTC 2013 [08:09:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:10:43 UTC 2013 [08:11:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:11:52 UTC 2013 [08:12:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:12:54 UTC 2013 [08:13:41] PROBLEM - Packetloss_Average on analytics1004 is CRITICAL: CRITICAL: packet_loss_average is 13.4427527132 (gt 8.0) [08:13:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:13:49 UTC 2013 [08:14:21] PROBLEM - Packetloss_Average on analytics1006 is CRITICAL: CRITICAL: packet_loss_average is 14.2994421875 (gt 8.0) [08:14:22] PROBLEM - Packetloss_Average on analytics1003 is CRITICAL: CRITICAL: packet_loss_average is 14.2830202326 (gt 8.0) [08:14:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:21] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:15:19 UTC 2013 [08:15:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 08:15:54 UTC 2013 [08:16:51] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:41] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 12.3314218692 (gt 8.0) [08:17:42] PROBLEM - Packetloss_Average on analytics1008 is CRITICAL: CRITICAL: packet_loss_average is 13.6796629231 (gt 8.0) [08:17:43] RECOVERY - Packetloss_Average on analytics1004 is OK: OK: packet_loss_average is 1.98619840336 [08:18:21] RECOVERY - Packetloss_Average on analytics1006 is OK: OK: packet_loss_average is 0.957937863248 [08:18:22] RECOVERY - Packetloss_Average on analytics1003 is OK: OK: packet_loss_average is 1.1069312605 [08:21:42] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is 0.207505666667 [08:21:42] RECOVERY - Packetloss_Average on analytics1008 is OK: OK: packet_loss_average is -0.0916690265487 [08:23:11] PROBLEM - Packetloss_Average on analytics1005 is CRITICAL: CRITICAL: packet_loss_average is 12.8281752672 (gt 8.0) [08:24:51] PROBLEM - Packetloss_Average on oxygen is CRITICAL: CRITICAL: packet_loss_average is 13.362825 (gt 8.0) [08:26:50] hi legoktm [08:27:11] RECOVERY - Packetloss_Average on analytics1005 is OK: OK: packet_loss_average is 0.0394880869565 [08:27:11] I'm finally done :D [08:27:51] So, does the log bot still need work? You already fixed most of it :P [08:28:51] RECOVERY - Packetloss_Average on oxygen is OK: OK: packet_loss_average is -0.19316745614 [08:31:14] oh, right! Congrats! [08:31:46] I think it would benefit from being drastically simplified, yes [08:32:55] I think the thing it needs most urgently is for someone else to validate that my patch does indeed make the bot recover well from disconnects [08:34:51] for testing purposes, IRC lib ships with a small, self-contained IRC server. You could try running a server in one terminal session and have the bot connect to it from the another. Then kill the server, wait a minute, and start it back up, and see if the bot manages to recover. [08:35:28] it's not very sexy work, but a lot of network programming is like that [08:39:49] ok, ill try and do that tomorrow then [08:48:51] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [08:48:58] cool, thank you. [10:20:29] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [10:21:29] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [11:12:32] New patchset: Petrb; "inserted 2 more packages needed to compile" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63065 [11:12:32] New patchset: Petrb; "create a link to shared folder" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63271 [11:22:47] were there any changes to rc irc recently? [11:26:37] Nemo_bis: BTW, it's not only the plwiki link in UW that's broken. [11:26:43] I just noticed other languages are broken as well [11:26:55] [you know, the thing I asked you about a few days ago…] [11:37:51] odder: yes, I remember [11:54:23] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:54:23] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:54:23] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:07:54 UTC 2013 [12:08:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:09:04 UTC 2013 [12:10:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:10:07 UTC 2013 [12:10:53] odder: hmpf, can't find any possibly related change [12:11:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:11:06 UTC 2013 [12:11:12] just file a bug and say precisely where the link is shown [12:11:27] also, this conversation should be on #wikimedia-tech, we're OT here [12:12:13] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:12:41 UTC 2013 [12:13:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:20] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:13:15 UTC 2013 [12:14:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 12:14:48 UTC 2013 [12:15:10] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:36] who cares. [13:24:54] !log restarted Jenkins. Got it by a memory bug again. [13:26:20] bah log bot is dead [13:26:45] nothing ever works [15:08:12] New review: coren; "I'm not a fan of polluting the root, but I agree that this is a reasonable convenience symlink." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/63271 [15:09:27] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63065 [15:09:35] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63271 [15:43:41] New patchset: Petrb; "removed the libxml because it breaks puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63279 [16:02:21] New patchset: coren; "Fix: libxml2-dev was included twice" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63279 [16:02:58] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63279 [16:07:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:07:51 UTC 2013 [16:08:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:08:58 UTC 2013 [16:09:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:09:59 UTC 2013 [16:10:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:10:54 UTC 2013 [16:11:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:11:42 UTC 2013 [16:12:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:12:58 UTC 2013 [16:13:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 16:15:04 UTC 2013 [16:15:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:09] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [16:34:10] PROBLEM - Puppet freshness on db26 is CRITICAL: No successful Puppet run in the last 10 hours [18:22:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [18:49:22] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [19:12:29] New patchset: coren; "Tool Labs: Also MOTD for the exec nodes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63291 [19:13:51] New review: coren; "Just cosmetic" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/63291 [19:13:52] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63291 [19:38:57] New patchset: coren; "Add locales to Tool Labs (also add fi_FI)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63308 [19:40:46] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63308 [20:08:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:07:56 UTC 2013 [20:08:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:09:03 UTC 2013 [20:09:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:10:04 UTC 2013 [20:10:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:10:59 UTC 2013 [20:11:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:11:47 UTC 2013 [20:12:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:13:02 UTC 2013 [20:13:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 11 20:14:48 UTC 2013 [20:15:50] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [20:21:10] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [20:22:10] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [21:01:35] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [21:02:35] PROBLEM - SSH on caesium is CRITICAL: Server answer: [21:02:35] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [21:04:35] RECOVERY - SSH on caesium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [21:29:05] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [21:29:54] RECOVERY - DPKG on mc15 is OK: All packages OK [21:55:04] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:55:04] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [21:55:04] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [23:04:32] New patchset: coren; "Tool Labs: More puppetized, part deux!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63348 [23:05:46] New patchset: coren; "Tool Labs: More puppetized, part deux!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63348 [23:07:57] New review: coren; "With great justice!" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/63348 [23:07:58] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63348 [23:08:14] PROBLEM - Host mw1173 is DOWN: PING CRITICAL - Packet loss = 100% [23:13:37] New patchset: coren; "Tool Labs: Fix syntax error in local.list" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63349 [23:15:15] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63349 [23:57:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time