[01:32:34] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.01775324345 secs [01:34:56] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.01705110073 secs [02:01:36] !log LocalisationUpdate completed (1.22wmf5) at Mon Jun 3 02:01:35 UTC 2013 [02:01:45] Logged the message, Master [02:02:19] !log LocalisationUpdate completed (1.22wmf4) at Mon Jun 3 02:02:19 UTC 2013 [02:02:29] Logged the message, Master [02:07:36] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 3 02:07:36 UTC 2013 [02:07:46] Logged the message, Master [04:31:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [04:40:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:41:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [05:00:26] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [05:02:36] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:18:48] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:19:39] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [05:27:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:28:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.198 second response time [06:21:43] PROBLEM - Host wtp1008 is DOWN: PING CRITICAL - Packet loss = 100% [06:22:03] RECOVERY - Host wtp1008 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [07:53:26] PROBLEM - Puppet freshness on db1032 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:26] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [07:53:26] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:26] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:26] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:26] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:26] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:27] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:27] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:28] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:28] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:29] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:29] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [08:01:46] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.004249453545 secs [08:02:36] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.008611083031 secs [10:50:16] New patchset: Akosiaris; "Adding packages sqoop, hbase, hive, pig to apt" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66397 [10:55:28] Change abandoned: Akosiaris; "Should avoid a new branch. Maybe even update this? https://www.mediawiki.org/wiki/Git/git-review#Sub..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66397 [11:00:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:58] New patchset: Akosiaris; "Adding packages sqoop, hbase, hive, pig to apt" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66398 [11:02:20] Change merged: Akosiaris; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66398 [11:02:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [11:54:16] mwscript extensions/TimedMediaHandler/maintenance/retryTranscodes.php --wiki commonswiki --error "av_interleaved_write_frame(): Invalid argument" [11:54:21] New review: ArielGlenn; "You really want one ChangeId in your commit message, and that would be from the first changeset you ..." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/64095 [11:54:21] that would be what j^ wants, right? [11:54:49] paravoid: yes [11:55:00] hi there :) [11:55:06] I wasn't sure if you were around [11:55:36] hi, wonders of irc ping [12:09:39] paravoid: thakns, looks like it worked, new encodes coming in [12:10:43] perfect [13:04:03] New patchset: Mark Bergsma; "Increase backend_weight from 20 to 100, to improve chash distribution" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66541 [13:05:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66541 [13:29:35] New patchset: Mark Bergsma; "Revert "Update the ldap scripts to pep8 compliant"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66542 [13:30:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66542 [13:31:20] New patchset: Mark Bergsma; "Revert "Fix various noc files to be pep8 compliant"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66543 [13:31:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66543 [13:36:27] akosiaris: aptmethod error receiving 'http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/dists/precise-cdh4/InRelease': [13:36:30] '404 Not Found' [13:41:20] akosiaris: yeah i noticed that too... I think that the cloudera repo should provide one but it does not... [13:41:25] paravoid: ^^ [13:41:34] talking to yourself? :P [13:41:36] * akosiaris stupid me :P [13:41:53] that's just InRelease I guess [13:42:05] the signed inline release file [13:42:07] akosiaris: I just merged your reprepro change on sockpuppet [13:42:17] after I reverted the other stuff that wasn't merged there [13:43:14] mark: thanks. I did not feel like running the puppet-merge with all the other stuff waiting to be merge too. I was trying to figure out whos' it was and pester him [13:43:23] yeah [13:44:36] akosiaris: nice thanks for the debianization stuff! i'm excited to try it out [13:44:50] i was looking, afaict right now it just installs the kafka .jar(s), right? [13:44:55] none of the bin/ scripts? [13:45:22] ottomata: exactly [13:45:29] that's what i am working on right now [13:45:55] ok cool, paravoid wanted me to avoid installing that myriad of scripts in /usr/bin, so if you're working it now [13:46:04] could you use my single kafka.sh wrapper instead of all of those? [13:46:13] and if you like, adapt it for any changes in 0.8? [13:46:15] (I can work on that later too) [13:46:37] https://gerrit.wikimedia.org/r/#/c/53170/10/debian/kafka.sh [13:46:42] send it to me and I will what i can do [13:46:50] ok cool. I 'll have a look into it [13:46:53] ahh, i'll email you 3 changes we should bring over from the 0.7.2 attempt [13:46:54] there are 2 more [13:47:32] ottomata: thanks for reaching out to twitter btw [13:47:59] and again, I don't mind not going through packaging/debianization if we can find other good ways [13:51:34] aye, ja! [13:51:42] welp, i mean, if akosiaris' stuff works, that is good enough for me [13:52:00] he's already done the work on it thus far. we might have to revisit this issue when we get around to doing storm [13:52:12] but at least in storm's case there's no scala/sbt to deal with [13:52:39] thank god [13:53:40] File "pool/main/b/bigtop-jsvc/bigtop-jsvc_1.0.10.orig.tar.gz" is already registered with different checksums! [13:53:40] md5 expected: c09da51e99fce3e6d8415ed8888ddcc6, got: ed33f68d0478c9afbbf33073be388c09 [13:53:48] nice [13:53:52] god damn ... what did those cloudera guys did ? [13:54:05] * akosiaris will figure it out [13:54:06] inserted a CIA backdoor in their code? [13:54:09] lol [13:54:26] yeah what a perfect example why just downloading tarballs without verification is a horrible idea ;) [13:55:09] ori had a similar idea for python modules which would be to set up our own pip mirror [13:55:19] but then, it is not that hard to package a python module. [13:55:23] right [13:55:38] the trouble of setting up and maintaining a pip mirror is probably more than actually packaging up those few modules that we need [13:55:45] yup [13:55:53] New patchset: Cmjohnson; "Adding new mac address ms-be1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66545 [13:56:07] next thing: package up mediawiki extensions *grin* [13:56:50] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66545 [13:59:04] ottomata: so, zookeeper? [13:59:07] paravoid: is it ok to assume that upstream tag versions have no correlation with debian tag versions ? [13:59:15] average: what do you mean? [13:59:42] oh, paravoid, yeah thanks for that review [13:59:43] paravoid: let's say upstream has 0.1 , 0.2 , 0.3 . Is it ok that debian packages will be for example 0.4 , 0.5 , 0.6 ? [13:59:44] upstream would have a "v1.0", debian would have "debian/1.0-1" [13:59:46] haven't got to that one yet [13:59:47] but [13:59:53] yes, we can make zookeeper its own module [13:59:54] i'm for that [13:59:55] no [14:00:04] (no?) [14:00:10] that was to average :) [14:00:11] sorry [14:00:13] oh ha, ok [14:00:27] yeah, i just have to check about where debian's zookeeper package keeps its config files [14:00:41] cdh4 has a /etc//conf convention [14:00:57] i betcha deb/ubu's has just /etc/zookeeper/ [14:01:05] but it should be doable [14:01:06] paravoid: the problem is that whenever I do git-dch --release , the new version is like "2.0.13ubuntu1" [14:01:25] paravoid: how can I avoid that ? I am forced to do manually --new-release="2.0.14" [14:01:33] which doesn't seem like a good idea [14:03:00] that's fine [14:03:18] the Debian package for upstream 0.1 should have a version of 0.1-1 [14:03:39] and subsequent packages for the same upstream due to packaging changes would be 0.1-2, -3 etc.) [14:04:30] Change merged: Ottomata; [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/66337 [14:05:59] Does WMF have any guidelines on deploying java internally? I know we have a few things deployed with Java and I was wondering if we have some document with information about stuff like how it should run, where logs should go, what to do if it is crashing, etc. [14:06:19] aahhhhhh hahah [14:06:26] manybubbles, welcome to the land of fun! [14:06:45] so, no? [14:06:51] "You want to deploy java?" => "lolno" [14:07:27] i mean, no, but i have been struggling with this issue for a few months now myself? there are no WMF guidelines so much, but afaict we try to stick to debian guidelines. /usr/share/java for .jars, etc. [14:07:50] but now i'm curious, whatcha working on? :) [14:08:34] me? I'll be working on search [14:08:42] ottomata: Manybubbles is the new Hero of Search! [14:09:11] but I also saw an email about a Suggester which looks like Java too. [14:09:11] excellent! [14:09:13] ah right, so solr / lucene stuff? [14:09:19] first assignment: rewrite to non-java ;-) [14:09:22] haha [14:09:31] :) [14:09:39] and not C# either hehe [14:09:46] yeah - solr/lucene stuff [14:10:08] and also not ruby [14:10:09] there is a lucene reimplemented in C... [14:10:10] aye. i think the answer you are going to get is: build a .deb that does stuff the debian way [14:10:18] and use that to deploy :) [14:10:38] ottomata: ;-) [14:11:05] ottomata: sure. but there is more to it then that, fortunately and unfortunately. [14:11:08] mark: Fun quiz. If it were rewritten in perl, would your (quite justified) hatred of JVM/.NET win over your (more disputable) loathing of perl? :-) [14:11:08] there is already a debian package for solr in apache bigtop [14:11:23] paravoid: so basically I have to use --new-version right ? [14:11:26] i don't hate perl [14:11:34] we just didn't standardize on it [14:11:42] so if it would be rewritten in perl, something would be wrong ;) [14:12:22] there's a Lucy module on CPAN [14:12:30] it works(have used it) [14:12:50] Lucy is a light-weight Lucene implementation in C [14:13:05] average: that is what I was thinking of! [14:13:15] manybubbles: :) [14:13:17] I think, thought, we'll probably want Solr [14:13:29] with all the bells and whistles. [14:14:37] and the thing with the jvm stuff is that there are a lot of extra ways you should be monitoring it and lots of fun stuff you can do when it goes sideways to generate forensic data [14:15:41] I"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." -- A. Saint-Exupery [14:15:46] I'm gonna drop a quick quote here.. [14:15:47] :) [14:16:04] paravoid: may I ask about the --new-version ? is it possible to avoid it ? [14:16:53] paravoid: I mean, I understand that if upstream is 0.2 , then the debian tag must be 0.2-\d , but can it do that automatically or does it have to be done manually with --new-version [14:17:02] ? [14:18:32] I don't use git-dch [14:18:34] just dch [14:18:52] ok [14:18:56] thanks :) [14:19:03] git-dch is just a tool to compile a changelog from git commits [14:19:10] it doesn't always make sense to use it [14:19:26] and you should certainly pick the version yourself, it just generates an example that works sometimes [14:19:35] (but not in our case, as it appends 'ubuntu') [14:19:55] I briefly looked at changing that ubuntu thing for package builder hosts once, but it wasn't trivial I think [14:24:07] PROBLEM - Puppet freshness on mw1149 is CRITICAL: No successful Puppet run in the last 10 hours [14:24:23] somebody at cloudera has really not understood debian packaging... [14:24:36] so... i got a file... bigtop-jsvc_1.0.10.orig.tar.gz [14:25:05] at Apr 22 it contained a single dir bigtop-jsvc-1.0.10-cdh4.2.1 and at May 28 it contained a single dir bigtop-jsvc-1.0.10-cdh4.3.0 [14:25:07] PROBLEM - Puppet freshness on mw1028 is CRITICAL: No successful Puppet run in the last 10 hours [14:25:21] dafuq ? [14:25:26] lol [14:29:07] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [14:33:37] RECOVERY - Host ms-be1 is UP: PING OK - Packet loss = 0%, RTA = 27.00 ms [14:33:53] hmm, akosiaris, may 28 cdh 4.3 was released, looks like they just updated their directories to match the version…not sure why they wouldn't have the tarball name change too [14:35:47] PROBLEM - swift-container-updater on ms-be1 is CRITICAL: Timeout while attempting connection [14:35:47] PROBLEM - swift-container-server on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:07] PROBLEM - DPKG on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:07] PROBLEM - swift-object-auditor on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:18] PROBLEM - Disk space on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:19] PROBLEM - swift-object-replicator on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:19] PROBLEM - swift-account-replicator on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:27] PROBLEM - RAID on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:37] PROBLEM - swift-object-updater on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:37] PROBLEM - swift-account-auditor on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:37] PROBLEM - swift-container-replicator on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:37] PROBLEM - SSH on ms-be1 is CRITICAL: Connection timed out [14:36:37] PROBLEM - swift-account-server on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:38] PROBLEM - swift-object-server on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:38] PROBLEM - swift-account-reaper on ms-be1 is CRITICAL: Timeout while attempting connection [14:36:47] PROBLEM - swift-container-auditor on ms-be1 is CRITICAL: Timeout while attempting connection [14:37:07] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [14:37:07] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [14:37:07] PROBLEM - Puppet freshness on amssq40 is CRITICAL: No successful Puppet run in the last 10 hours [14:37:07] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [14:37:07] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:10] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:10] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:10] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:10] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:10] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [14:40:41] ottomata: the .orig should only have the upstream tarball. Not their own changes inside. [14:41:28] ottomata: hive, sqoop, pig and rest debs are in apt [14:42:56] danke! at 4.2.1? [14:43:47] nope. 4.3 [14:44:22] do you want 4.2.1 or care to upgrade ? [14:44:56] upgrade is cool, but i think we have to do it all at once [14:44:58] so we'd need all of them [14:45:31] i was getting some dependency issue when trying hive 4.3.0 without hbase 4.3.0 etc. [14:45:50] that should be fixed now. both are at 4.3.0 [14:46:04] oh hmm, true, hm, ok, [14:46:06] i guess that's fine [14:46:08] hm [14:46:20] i'd rather run all cdh packages at the same version [14:46:22] but it is probably fine [14:46:27] including hadoop etc. [14:46:38] those are at 4.3.0 too at the repo [14:46:41] oh! [14:46:41] ok [14:46:43] cool [14:46:44] danke [14:46:47] then that should be fine i think [14:47:06] i will upgrade that stuff before I try to apply the new puppetization to a hadoop node then [14:47:06] thank you [14:47:11] bitte schon [14:48:21] PROBLEM - NTP on ms-be1 is CRITICAL: NTP CRITICAL: No response from NTP server [14:51:07] akosiaris: are the 4.2.1 hadoop .debs still in apt? [14:51:22] ah, no they aren't [14:51:22] hm [14:51:26] is there a way to pass dch parameters from git-dch ? [14:51:31] like for example --force-distribution [14:55:51] PROBLEM - Host ms-be1 is DOWN: PING CRITICAL - Packet loss = 100% [14:57:24] no idea :) [15:00:30] RECOVERY - SSH on ms-be1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [15:00:37] * mark upgrades his Eclipse install [15:00:45] RECOVERY - Host ms-be1 is UP: PING OK - Packet loss = 0%, RTA = 26.61 ms [15:00:49] the whaaaat? [15:01:19] I liked eclipse a lot when I was programming in Java [15:01:22] is that yet another python script on our cluster? [15:04:03] are you thinking I'm getting old again? ;) [15:04:26] mark: Java is a kind of crummy language so it needs a good IDE. [15:04:36] it does [15:18:31] New patchset: Andrew Bogott; "Add a custom job that runs pep8 on each .py file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66548 [15:18:47] hashar: there's the custom script [15:19:01] now to figure out why I can't submit a review to jenkins-job-builder [15:33:29] Change abandoned: Andrew Bogott; "Moved this file to the integration/jenkins repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66548 [15:34:32] paravoid: what does this say on your machine ? git-buildpackage --version [15:34:47] which machine? :) [15:35:02] I use Debian stable on my desktop, Debian unstable on my build box and I have a precise chroot for wikimedia work [15:35:46] paravoid: ok, it seems that git-buildpackage has some new stuff that I'd like to use [15:36:17] paravoid: for example the -D switch came in april 2013 [15:37:07] paravoid: can we use the new one ? [15:37:42] paravoid: because you probably have a different version than I have (I'm using ubuntu raring) [15:37:44] New patchset: Ottomata; "Puppetizing analytics1020 with roles/hadoop.pp!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66553 [15:38:29] New review: Ottomata; "Faidon, I'm adding you as a reviewer mainly to get feedback on the new roles/hadoop.pp file." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66553 [15:39:35] New patchset: Ottomata; "Puppetizing analytics1020 with roles/hadoop.pp!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66553 [15:45:28] @notify binasher [15:45:29] I'll let you know when I see binasher around here [15:47:51] what's -D? [15:48:16] average: ^ [15:49:05] paravoid: --distribution [15:49:17] paravoid: it's like in the changelog when you see "raring" or "precise" [15:49:22] paravoid: that one you can set with -D [15:49:36] is that git-buildpackage? [15:49:36] paravoid: I need that because our packages need to be with "wikimedia" in that field [15:49:42] paravoid: yes it is, but a newer version [15:49:47] you know you can just vi debian/changelog, right? :) [15:49:54] paravoid: yes but.. [15:50:09] or just use dch directly [15:51:00] remember that if there were no human involvement required, we didn't need you to build the packages in the first place ;-) [15:52:57] alright [15:59:00] mark , paravoid should you change your mind http://garage-coding.com/releases/git-buildpackage/ [15:59:10] in the meantime I'll edit the distribution manually [15:59:36] feel free to use whatever git-buildpackage version you want [15:59:56] as long as it can be kept compatible with previous versions, e.g. no newer incompatible gbp.conf options [16:00:46] oh cool :) [16:00:47] thanks [16:06:31] mark, did you object to the patch you reverted here https://gerrit.wikimedia.org/r/#/c/66542/ or were you just backing it out because it wasn't on sockpuppet? (If the latter, I will re-submit and merge now.) [16:06:41] * Jeff_Green wonders about tridge:/data @ %100 [16:06:48] just the latter [16:06:51] you can remerge any time [16:06:58] as long as you also merge on sockpuppet and don't block others :) [16:08:16] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Mon Jun 3 16:08:11 UTC 2013 [16:08:16] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Mon Jun 3 16:08:11 UTC 2013 [16:08:16] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Mon Jun 3 16:08:12 UTC 2013 [16:08:16] RECOVERY - Puppet freshness on mc16 is OK: puppet ran at Mon Jun 3 16:08:13 UTC 2013 [16:08:16] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Mon Jun 3 16:08:13 UTC 2013 [16:08:17] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Mon Jun 3 16:08:14 UTC 2013 [16:08:17] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Mon Jun 3 16:08:14 UTC 2013 [16:08:18] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Mon Jun 3 16:08:15 UTC 2013 [16:08:18] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Mon Jun 3 16:08:15 UTC 2013 [16:08:26] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Mon Jun 3 16:08:16 UTC 2013 [16:08:26] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Mon Jun 3 16:08:16 UTC 2013 [16:08:26] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Mon Jun 3 16:08:17 UTC 2013 [16:08:26] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Mon Jun 3 16:08:17 UTC 2013 [16:08:26] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Mon Jun 3 16:08:18 UTC 2013 [16:08:27] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Mon Jun 3 16:08:18 UTC 2013 [16:08:27] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Mon Jun 3 16:08:19 UTC 2013 [16:08:28] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Mon Jun 3 16:08:19 UTC 2013 [16:08:28] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Mon Jun 3 16:08:20 UTC 2013 [16:08:29] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Mon Jun 3 16:08:20 UTC 2013 [16:08:29] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Mon Jun 3 16:08:21 UTC 2013 [16:08:30] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Mon Jun 3 16:08:22 UTC 2013 [16:08:30] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Mon Jun 3 16:08:22 UTC 2013 [16:08:31] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Mon Jun 3 16:08:23 UTC 2013 [16:08:31] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Mon Jun 3 16:08:23 UTC 2013 [16:08:32] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Mon Jun 3 16:08:23 UTC 2013 [16:08:32] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Mon Jun 3 16:08:24 UTC 2013 [16:08:33] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Mon Jun 3 16:08:24 UTC 2013 [16:08:33] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Mon Jun 3 16:08:25 UTC 2013 [16:08:34] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Mon Jun 3 16:08:25 UTC 2013 [16:08:34] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Mon Jun 3 16:08:25 UTC 2013 [16:08:36] RECOVERY - Puppet freshness on mc2 is OK: puppet ran at Mon Jun 3 16:08:27 UTC 2013 [16:08:36] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Mon Jun 3 16:08:27 UTC 2013 [16:08:36] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Mon Jun 3 16:08:28 UTC 2013 [16:08:36] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Mon Jun 3 16:08:28 UTC 2013 [16:08:37] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Mon Jun 3 16:08:29 UTC 2013 [16:08:37] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Mon Jun 3 16:08:30 UTC 2013 [16:08:38] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Mon Jun 3 16:08:30 UTC 2013 [16:08:38] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Mon Jun 3 16:08:31 UTC 2013 [16:08:39] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Mon Jun 3 16:08:32 UTC 2013 [16:08:39] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Mon Jun 3 16:08:32 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Mon Jun 3 16:08:33 UTC 2013 [16:08:40] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Mon Jun 3 16:08:33 UTC 2013 [16:08:41] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Mon Jun 3 16:08:33 UTC 2013 [16:08:41] RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Mon Jun 3 16:08:34 UTC 2013 [16:08:42] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Mon Jun 3 16:08:34 UTC 2013 [16:08:42] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Mon Jun 3 16:08:35 UTC 2013 [16:08:46] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Mon Jun 3 16:08:35 UTC 2013 [16:08:46] RECOVERY - Puppet freshness on professor is OK: puppet ran at Mon Jun 3 16:08:36 UTC 2013 [16:08:46] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Mon Jun 3 16:08:37 UTC 2013 [16:08:46] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Mon Jun 3 16:08:38 UTC 2013 [16:08:46] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Mon Jun 3 16:08:39 UTC 2013 [16:08:47] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Mon Jun 3 16:08:41 UTC 2013 [16:08:47] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Mon Jun 3 16:08:42 UTC 2013 [16:08:48] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Mon Jun 3 16:08:43 UTC 2013 [16:08:48] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Mon Jun 3 16:08:43 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Mon Jun 3 16:08:43 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Mon Jun 3 16:08:43 UTC 2013 [16:08:50] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Mon Jun 3 16:08:44 UTC 2013 [16:08:50] RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Mon Jun 3 16:08:44 UTC 2013 [16:08:51] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Mon Jun 3 16:08:44 UTC 2013 [16:08:56] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Mon Jun 3 16:08:46 UTC 2013 [16:08:56] RECOVERY - Puppet freshness on db73 is OK: puppet ran at Mon Jun 3 16:08:47 UTC 2013 [16:08:56] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Mon Jun 3 16:08:48 UTC 2013 [16:08:56] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Mon Jun 3 16:08:48 UTC 2013 [16:08:56] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Mon Jun 3 16:08:49 UTC 2013 [16:08:57] RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Mon Jun 3 16:08:49 UTC 2013 [16:08:57] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Mon Jun 3 16:08:50 UTC 2013 [16:08:58] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Mon Jun 3 16:08:50 UTC 2013 [16:08:58] RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Mon Jun 3 16:08:51 UTC 2013 [16:08:59] RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Mon Jun 3 16:08:51 UTC 2013 [16:08:59] RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Mon Jun 3 16:08:52 UTC 2013 [16:09:00] RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Mon Jun 3 16:08:52 UTC 2013 [16:09:00] RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Mon Jun 3 16:08:53 UTC 2013 [16:09:01] RECOVERY - Puppet freshness on tmh1 is OK: puppet ran at Mon Jun 3 16:08:54 UTC 2013 [16:09:01] RECOVERY - Puppet freshness on mc8 is OK: puppet ran at Mon Jun 3 16:08:54 UTC 2013 [16:09:02] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Mon Jun 3 16:08:55 UTC 2013 [16:09:06] RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Mon Jun 3 16:08:56 UTC 2013 [16:09:06] RECOVERY - Puppet freshness on db60 is OK: puppet ran at Mon Jun 3 16:08:57 UTC 2013 [16:09:06] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Mon Jun 3 16:08:57 UTC 2013 [16:09:06] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Mon Jun 3 16:08:57 UTC 2013 [16:09:06] RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Mon Jun 3 16:08:58 UTC 2013 [16:09:07] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Mon Jun 3 16:08:59 UTC 2013 [16:09:07] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Mon Jun 3 16:08:59 UTC 2013 [16:09:08] RECOVERY - Puppet freshness on linne is OK: puppet ran at Mon Jun 3 16:09:00 UTC 2013 [16:09:08] RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Mon Jun 3 16:09:00 UTC 2013 [16:09:09] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Mon Jun 3 16:09:00 UTC 2013 [16:09:09] RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Mon Jun 3 16:09:04 UTC 2013 [16:09:10] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Mon Jun 3 16:09:04 UTC 2013 [16:09:10] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Mon Jun 3 16:09:04 UTC 2013 [16:09:11] RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Mon Jun 3 16:09:04 UTC 2013 [16:09:11] RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Mon Jun 3 16:09:04 UTC 2013 [16:09:12] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Mon Jun 3 16:09:05 UTC 2013 [16:09:12] RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Mon Jun 3 16:09:05 UTC 2013 [16:09:16] RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Mon Jun 3 16:09:07 UTC 2013 [16:09:16] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Mon Jun 3 16:09:07 UTC 2013 [16:09:16] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Mon Jun 3 16:09:09 UTC 2013 [16:09:16] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Mon Jun 3 16:09:10 UTC 2013 [16:09:16] RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Mon Jun 3 16:09:11 UTC 2013 [16:09:17] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Mon Jun 3 16:09:11 UTC 2013 [16:09:17] RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Mon Jun 3 16:09:12 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Mon Jun 3 16:09:12 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Mon Jun 3 16:09:13 UTC 2013 [16:09:19] RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Mon Jun 3 16:09:15 UTC 2013 [16:09:19]