[00:01:52] mutante: and if you are really brave I can really use https://gerrit.wikimedia.org/r/#/c/82625/ which would setup a (fire walled) git daemon on gallium =D Tested it on labs and prod already. [00:03:34] I hate puppet [00:07:57] (03PS1) 10Hashar: zuul: missing column in template variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/83593 [00:09:02] can I get https://gerrit.wikimedia.org/r/#/c/83593/ merged please? That fix up a typo thx [00:09:48] mutante: but had a typo :( [00:11:07] (03PS10) 10Hashar: contint: publish Zuul git over git protocol [operations/puppet] - 10https://gerrit.wikimedia.org/r/82625 [00:18:47] (03CR) 10Dzahn: [C: 032] zuul: missing column in template variable [operations/puppet] - 10https://gerrit.wikimedia.org/r/83593 (owner: 10Hashar) [00:19:24] s/column/colon [00:20:41] arh [00:21:09] not colon mutante! http://en.wikipedia.org/wiki/Colon_(anatomy) [00:21:11] :D [00:21:30] !log csteipp synchronized php-1.22wmf15/extensions/CentralAuth 'Syncing correct submodule checkout of centralauth' [00:21:33] Logged the message, Master [00:22:08] (03CR) 10Hashar: "rebaed" [operations/puppet] - 10https://gerrit.wikimedia.org/r/82625 (owner: 10Hashar) [00:22:08] PROBLEM - Disk space on cp1059 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 10549 MB (3% inode=99%): /srv/sdb3 11419 MB (3% inode=99%): [00:22:08] PROBLEM - search indices - check lucene status page on search1019 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 59205 bytes in 0.015 second response time [00:22:09] PROBLEM - Disk space on es1003 is CRITICAL: DISK CRITICAL - free space: /a 302148 MB (3% inode=99%): [00:22:23] cp1059 isn't happy [00:23:11] there is a bunch of cp** hosts having the same issue [00:23:38] There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused [00:24:48] PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 68 statistics [00:24:58] LeslieCarr: ^ Can you kick ganglia please? [00:25:15] Reedy: on whcih host ? [00:25:27] http://ganglia.wikimedia.org/latest/ [00:25:30] (03CR) 10Hashar: "That fixed up the issue. /etc/default/zuul is now empty :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83593 (owner: 10Hashar) [00:25:33] ah [00:26:48] RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 7 statistics [00:26:57] restarted [00:27:03] !log restarted gmetad on nickel [00:27:06] Logged the message, Mistress of the network gear. [00:28:17] Mostlinkedtemplates on commonswiki is an expensive on among the special page updates [00:30:19] Still down.. [00:33:49] PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 84 statistics [00:33:52] LeslieCarr: Web interface is still showing the same error [00:34:48] RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 2 statistics [00:35:15] it listens on 8649 but not on 8652 [00:35:21] eh, 8654 [00:35:36] 8654 is configured as "interactive_port" [00:35:43] :/ [00:40:00] !log restarting gmetad on nickel [00:40:04] Logged the message, Master [00:40:54] Starting Ganglia Monitor Meta-Daemon: Warning: we failed to resolve data source name sq31.wikimedia.org [00:40:57] hrmm [00:41:26] that's just a warn, so whats the deal [00:41:45] gmetad is the one that should listen 8654 [00:42:05] gmetad[13746]: segfault at 18 bahh [00:44:04] https://lists.oetiker.ch/pipermail/rrd-users/2010-July.txt [00:44:24] rsync: write failed on "/mnt/ganglia_tmp/rrds.pmtpa/API application servers eqiad/mw1114.eqiad.wmnet/ap_busy_workers.rrd": No space left on device (28) [00:46:15] Reedy: out of disk, we're cleaning up some old files [00:46:24] heh [00:46:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:47] mutante: what's the size of rrds from 'client-side' [00:46:48] ? [00:48:08] ori-l: 4.0G /mnt/ganglia_tmp on nickel .. client-side? [00:48:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [00:48:47] ah, being told it's in memory [00:49:42] LeslieCarr just cleaned it up:) trying to start it again [00:50:13] Reedy: fixed [00:50:57] (03PS1) 10Akosiaris: lint manifests/misc/install-server.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/83598 [00:52:04] (03CR) 10Akosiaris: [C: 032] lint manifests/misc/install-server.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/83598 (owner: 10Akosiaris) [00:52:18] cya, afk'ing [00:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [00:54:27] sending update-special-pages to background on hume [00:54:30] out [00:57:39] mutante / LeslieCarr: thanks! [01:06:27] PROBLEM - Memcached on holmium is CRITICAL: Connection refused [01:06:27] PROBLEM - search indices - check lucene status page on search1019 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 59205 bytes in 0.015 second response time [01:06:27] PROBLEM - DPKG on stafford is CRITICAL: DPKG CRITICAL dpkg reports broken packages [01:06:27] PROBLEM - RAID on db1031 is CRITICAL: CRITICAL: Degraded [01:06:27] PROBLEM - Disk space on cp1059 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 10546 MB (3% inode=99%): /srv/sdb3 11417 MB (3% inode=99%): [01:34:19] RoanKattouw: Mind deleting a git repo? It was imported from SVN, but there's a newer version in another repo [01:34:21] ssh -p 29418 gerrit.wikimedia.org delete-project delete --yes-really-delete -- mediawiki/extensions/SlippyMap.git [01:34:36] newer being mediawiki/extensions/OpenStreetSlippyMap [01:37:35] lol @ --yes-really-delete [01:37:48] Cannot delete project mediawiki/extensions/SlippyMap: Project is subscribed by other projects. [01:38:02] There's --force... [01:38:04] subscribed? [01:38:13] I wonder if it means mediawiki/extensions.git [01:38:17] Reedy: submodule i would guess [01:38:17] I think this might be referring to --- yeah that [01:38:33] I'll merge my commit to remove it [01:39:22] And now it ran fine [01:39:41] Great [01:40:02] Tanks [01:40:03] *Thanks [02:09:57] (03PS1) 10Ori.livneh: Make defaults for StatsD module more sensible [operations/puppet] - 10https://gerrit.wikimedia.org/r/83602 [02:09:58] (03PS1) 10Ori.livneh: Provision generic StatsD instance on professor.pmtpa.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/83603 [05:28:24] ori-l: uhm, wasn't the issue with professor the fact that it's lucid? [05:32:15] i should really try sleeping more than four hours a night [05:32:24] yes, you're right. [05:32:41] gam ani [05:32:43] is there another host that would be suitable? [05:37:09] let's ask tomorrow [05:37:20] maybe we should start to move graphite [05:37:27] start moving [05:37:29] even [05:37:47] although this isn't the best week, looks very heavily meeting-oriented [05:38:12] i think it'd be a missed opportunity if we just moved it as-is without also reconsidering certain aspects [05:38:36] specifically: 1) a namespacing convention for metrics [05:38:37] like what? [05:38:51] bd808 devised one at his previous gig [05:39:05] and it's something that i've seen people mention often as essential to good organization [05:39:13] do you want to grab a few people tomorrow and talk about all that for 10-15' or so? [05:39:33] I really haven't worked with graphite before and don't know much about it [05:39:41] but I'd be happy to review stuff for you :) [05:39:57] and brainstorm [05:40:21] sure [05:41:54] the technical aspect of namespacing is pretty thin; it's all conceptual, a bit like the filesystem hierarchy standards. graphite metric names are more or less free-form but most UIs split on '.' [05:42:22] so you can do ..., but there are other schemes too [05:43:03] anyways: tomorrow. we should also consider if we want to keep all the metrics we're currently logging [05:43:52] i don't know if you've looked recently but it's a bit of a no-man's land; professor has ~400 gigs of metrics. i imagine only a dozen out of the several hundred metrics we log get looekd at [05:44:41] good night [05:44:53] lol [05:47:05] paravoid: i use graphite alot, if any help needed, let me know [05:55:34] matanya: i don't think there's a specific issue we need help with, but if you have some best practices to recommend, that would be awesome [05:58:45] ori-l: i can write ir down in mediawiki [05:59:11] i'd really appreciate that [05:59:17] i'm sure others would too [06:04:48] * bd808 reads scrollback [06:15:41] graphite metric naming convention was: company.environment.colo.host.app.metric. [06:16:57] I have a few notes on it with some detail and can probably talk reasonably intelligently about some of the graphs it let us create [06:18:06] Our first big step towards organization was to talk through "why are we collecting metrics" [06:24:46] bd808: As long as it makes pretty ones, the rest is fine [06:27:15] Having a consistent and properly granular hierarchy makes creating useful graphs much easier. [06:27:56] graphite has nice functions for rolling up metrics in aggregate values [06:28:46] so you can make a dashboard that shows performance of X across enterprise and then drill down into each colo, rack etc [06:29:45] our scheme didn't include rack in an easy to understand way which caused some annoyance eventually [08:02:53] (03CR) 10Jeroen De Dauw: "Huh. Damn you FireFox - I don't want this ig bullshit!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83348 (owner: 10Jeroen De Dauw) [08:03:46] (03PS3) 10Jeroen De Dauw: Update some rss/atom feeds in the planet wikimedia config [operations/puppet] - 10https://gerrit.wikimedia.org/r/83348 [08:03:59] (03PS1) 10TTO: Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 [08:04:47] (03PS4) 10Jeroen De Dauw: Update some rss/atom feeds in the planet wikimedia config [operations/puppet] - 10https://gerrit.wikimedia.org/r/83348 [09:36:15] (03PS1) 10Dereckson: Maintenance for http://fr.planet.wikimedia.org/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/83620 [11:27:01] RobH: around? :) [11:33:18] if possible it would be great to know whats going on here >> http://ganglia.wikimedia.org/latest/?r=2hr&cs=&ce=&m=cpu_report&s=by+name&c=Labs+NFS+cluster+pmtpa&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [12:20:37] addshore: that means beta is down doesn't it [12:20:58] yep, 503 [12:21:01] everything labs side thats uses that labstore nfs [12:21:02] !log rebooted neon, it was out to lunch [12:21:06] Logged the message, Master [12:21:16] whole of tools labs, beta etc ;p [12:21:31] yeah, but nmobody cares about tools [12:21:37] :P [12:21:44] it's not production, just something set up ion order to kill TS [12:21:47] ;) [12:22:37] would be nice to have labstore3 back ;p [12:22:59] !!log labstore3 overloaded, beta & tools &co. down [12:22:59] petan needs a new hobby :P [12:23:02] it just went into cpu overdrive at about 10:15 :P [12:23:04] !log labstore3 overloaded, beta & tools &co. down [12:23:07] Logged the message, Master [12:43:12] !log rebooting labstore3 after restart of nfsd did not do the trick [12:45:02] :> [12:46:16] not behaving [12:47:35] apergos: I notice :/ *watching ganglia* [12:49:04] and now I can't get nfsd to restart nor the nfs volume manager. awesome [12:49:17] did try to be orderly about shuttting them down first before power cycle too [12:49:18] grrr [13:01:18] :/ [13:05:01] no /mnt/srv [13:09:44] Sep 10 12:45:03 labstore3 mdadm[2234]: DeviceDisappeared event detected on md de [13:09:44] vice /dev/md/nfs0 and so on for the rest of the devices, then md: md114 stopped [13:10:11] but then a bunch of md/raid:md114: device md121 operational as raid disk 0 etc [13:18:26] so here's the story folks. the lines for mounting store-srv, store-repo etc are commented out in fstab from aug 15 [13:18:41] I can't find anything in the logs nor puppet to tell me what that's about [13:19:39] maybe it's ok to just uncomment them and mount everything; maybe Coren was doing work over there and I'd be breaking something [13:19:55] what I really need is Coren to have a look (sorry) [13:21:11] Okay:-) [13:24:55] !log on labstore3, the various mount lines for /dev/mapper/store-xx are commented out in fstab from aug 15, so naturally there's o nfs etc, not sure if it's ok to just put 'em back in, need someone to take a look-see [13:29:39] if he's in pst it will be a few more hours [13:43:01] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Tue Sep 10 13:42:56 UTC 2013 [13:43:01] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Tue Sep 10 13:42:56 UTC 2013 [13:43:01] RECOVERY - Puppet freshness on iodine is OK: puppet ran at Tue Sep 10 13:42:56 UTC 2013 [13:43:01] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Tue Sep 10 13:42:56 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Tue Sep 10 13:43:01 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Tue Sep 10 13:43:01 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Tue Sep 10 13:43:06 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Tue Sep 10 13:43:06 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Tue Sep 10 13:43:06 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Tue Sep 10 13:43:06 UTC 2013 [13:43:10] RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Tue Sep 10 13:43:06 UTC 2013 [13:43:11] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Tue Sep 10 13:43:06 UTC 2013 [13:43:20] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Tue Sep 10 13:43:11 UTC 2013 [13:43:50] RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Tue Sep 10 13:43:46 UTC 2013 [13:43:50] RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Tue Sep 10 13:43:47 UTC 2013 [13:44:10] RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Tue Sep 10 13:43:57 UTC 2013 [13:44:10] RECOVERY - Puppet freshness on search25 is OK: puppet ran at Tue Sep 10 13:44:02 UTC 2013 [13:44:10] RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Tue Sep 10 13:44:02 UTC 2013 [13:44:10] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Tue Sep 10 13:44:02 UTC 2013 [13:44:10] RECOVERY - Puppet freshness on db54 is OK: puppet ran at Tue Sep 10 13:44:02 UTC 2013 [13:45:19] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Tue Sep 10 13:45:12 UTC 2013 [13:45:19] RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Tue Sep 10 13:45:13 UTC 2013 [13:45:19] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Tue Sep 10 13:45:13 UTC 2013 [13:45:19] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Tue Sep 10 13:45:13 UTC 2013 [13:45:19] RECOVERY - Puppet freshness on mw1104 is OK: puppet ran at Tue Sep 10 13:45:13 UTC 2013 [13:45:20] RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Tue Sep 10 13:45:13 UTC 2013 [13:45:29] RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Tue Sep 10 13:45:23 UTC 2013 [13:45:59] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Tue Sep 10 13:45:48 UTC 2013 [13:45:59] RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Tue Sep 10 13:45:53 UTC 2013 [13:46:00] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Tue Sep 10 13:45:53 UTC 2013 [13:46:00] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Tue Sep 10 13:45:53 UTC 2013 [13:46:00] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:00] RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:00] RECOVERY - Puppet freshness on mw1021 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:01] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:01] RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:02] RECOVERY - Puppet freshness on lvs1004 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:02] RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:03] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:03] RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:04] RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:04] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:05] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:05] RECOVERY - Puppet freshness on cp1018 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:06] RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:06] RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:09] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:09] RECOVERY - Puppet freshness on capella is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:09] RECOVERY - Puppet freshness on mw1208 is OK: puppet ran at Tue Sep 10 13:45:58 UTC 2013 [13:46:09] RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Tue Sep 10 13:45:59 UTC 2013 [13:46:09] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Tue Sep 10 13:45:59 UTC 2013 [13:46:10] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Tue Sep 10 13:45:59 UTC 2013 [13:46:10] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Tue Sep 10 13:45:59 UTC 2013 [13:46:11] RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Tue Sep 10 13:45:59 UTC 2013 [13:46:11] RECOVERY - Puppet freshness on cp1060 is OK: puppet ran at Tue Sep 10 13:46:04 UTC 2013 [13:46:12] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Tue Sep 10 13:46:04 UTC 2013 [13:46:12] RECOVERY - Puppet freshness on cp1066 is OK: puppet ran at Tue Sep 10 13:46:04 UTC 2013 [13:46:13] RECOVERY - Puppet freshness on mw13 is OK: puppet ran at Tue Sep 10 13:46:04 UTC 2013 [13:46:13] RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Tue Sep 10 13:46:04 UTC 2013 [13:46:14] RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Tue Sep 10 13:46:04 UTC 2013 [13:46:21] RECOVERY - Puppet freshness on ssl1005 is OK: puppet ran at Tue Sep 10 13:46:09 UTC 2013 [13:46:22] RECOVERY - Puppet freshness on mw93 is OK: puppet ran at Tue Sep 10 13:46:09 UTC 2013 [13:46:22] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Tue Sep 10 13:46:09 UTC 2013 [13:46:22] RECOVERY - Puppet freshness on srv299 is OK: puppet ran at Tue Sep 10 13:46:09 UTC 2013 [13:46:22] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Tue Sep 10 13:46:14 UTC 2013 [13:46:22] RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Tue Sep 10 13:46:14 UTC 2013 [13:46:22] RECOVERY - Puppet freshness on mw1180 is OK: puppet ran at Tue Sep 10 13:46:14 UTC 2013 [13:46:23] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Tue Sep 10 13:46:14 UTC 2013 [13:46:29] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:29] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:29] RECOVERY - Puppet freshness on hume is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:29] RECOVERY - Puppet freshness on mw1128 is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:29] RECOVERY - Puppet freshness on mw1194 is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:30] RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:30] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Tue Sep 10 13:46:19 UTC 2013 [13:46:31] RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Tue Sep 10 13:46:24 UTC 2013 [13:46:31] RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Tue Sep 10 13:46:24 UTC 2013 [13:46:32] RECOVERY - Puppet freshness on tmh1001 is OK: puppet ran at Tue Sep 10 13:46:24 UTC 2013 [13:46:39] RECOVERY - Puppet freshness on sanger is OK: puppet ran at Tue Sep 10 13:46:34 UTC 2013 [13:46:39] RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Tue Sep 10 13:46:34 UTC 2013 [13:46:49] RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Tue Sep 10 13:46:39 UTC 2013 [13:46:49] RECOVERY - Puppet freshness on db69 is OK: puppet ran at Tue Sep 10 13:46:39 UTC 2013 [13:46:49] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Tue Sep 10 13:46:39 UTC 2013 [13:46:49] RECOVERY - Puppet freshness on hafnium is OK: puppet ran at Tue Sep 10 13:46:44 UTC 2013 [13:46:49] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Tue Sep 10 13:46:44 UTC 2013 [13:46:50] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Tue Sep 10 13:46:44 UTC 2013 [13:46:50] RECOVERY - Puppet freshness on db32 is OK: puppet ran at Tue Sep 10 13:46:44 UTC 2013 [13:46:51] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Tue Sep 10 13:46:44 UTC 2013 [13:46:51] RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Tue Sep 10 13:46:44 UTC 2013 [13:46:59] RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Tue Sep 10 13:46:54 UTC 2013 [13:46:59] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Tue Sep 10 13:46:54 UTC 2013 [13:46:59] RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Tue Sep 10 13:46:54 UTC 2013 [13:46:59] RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Tue Sep 10 13:46:54 UTC 2013 [13:46:59] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Tue Sep 10 13:46:54 UTC 2013 [13:47:09] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Tue Sep 10 13:46:59 UTC 2013 [13:47:09] RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Tue Sep 10 13:46:59 UTC 2013 [13:47:09] RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Tue Sep 10 13:47:04 UTC 2013 [13:47:09] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Tue Sep 10 13:47:04 UTC 2013 [13:47:09] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Tue Sep 10 13:47:04 UTC 2013 [13:47:10] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Tue Sep 10 13:47:04 UTC 2013 [13:47:19] RECOVERY - Puppet freshness on cp1062 is OK: puppet ran at Tue Sep 10 13:47:09 UTC 2013 [13:47:19] RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Tue Sep 10 13:47:14 UTC 2013 [13:47:29] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Tue Sep 10 13:47:19 UTC 2013 [13:47:29] RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Tue Sep 10 13:47:19 UTC 2013 [13:47:29] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Tue Sep 10 13:47:19 UTC 2013 [13:47:39] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Tue Sep 10 13:47:34 UTC 2013 [13:47:49] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Tue Sep 10 13:47:39 UTC 2013 [13:47:49] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Tue Sep 10 13:47:39 UTC 2013 [13:47:49] RECOVERY - Puppet freshness on db77 is OK: puppet ran at Tue Sep 10 13:47:39 UTC 2013 [13:47:49] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Tue Sep 10 13:47:39 UTC 2013 [13:47:49] RECOVERY - Puppet freshness on db65 is OK: puppet ran at Tue Sep 10 13:47:44 UTC 2013 [13:47:49] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Tue Sep 10 13:47:44 UTC 2013 [13:47:59] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Tue Sep 10 13:47:49 UTC 2013 [13:47:59] RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Tue Sep 10 13:47:49 UTC 2013 [13:47:59] RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Tue Sep 10 13:47:54 UTC 2013 [13:47:59] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Tue Sep 10 13:47:54 UTC 2013 [13:47:59] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Tue Sep 10 13:47:54 UTC 2013 [13:48:09] RECOVERY - Puppet freshness on erbium is OK: puppet ran at Tue Sep 10 13:47:59 UTC 2013 [13:48:09] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Tue Sep 10 13:47:59 UTC 2013 [13:48:09] RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Tue Sep 10 13:47:59 UTC 2013 [13:48:09] RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Tue Sep 10 13:47:59 UTC 2013 [13:48:09] RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Tue Sep 10 13:48:04 UTC 2013 [13:48:20] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Tue Sep 10 13:48:09 UTC 2013 [13:48:20] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Tue Sep 10 13:48:09 UTC 2013 [13:48:20] RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Tue Sep 10 13:48:09 UTC 2013 [13:48:20] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Tue Sep 10 13:48:14 UTC 2013 [13:48:20] RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Tue Sep 10 13:48:14 UTC 2013 [13:48:20] RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Tue Sep 10 13:48:14 UTC 2013 [13:48:29] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Tue Sep 10 13:48:19 UTC 2013 [13:48:29] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Tue Sep 10 13:48:19 UTC 2013 [13:48:29] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Tue Sep 10 13:48:19 UTC 2013 [13:48:41] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Tue Sep 10 13:48:29 UTC 2013 [13:48:41] RECOVERY - Puppet freshness on db57 is OK: puppet ran at Tue Sep 10 13:48:29 UTC 2013 [13:48:41] RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Tue Sep 10 13:48:29 UTC 2013 [13:48:41] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Tue Sep 10 13:48:34 UTC 2013 [13:48:41] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Tue Sep 10 13:48:34 UTC 2013 [13:48:42] RECOVERY - Puppet freshness on db51 is OK: puppet ran at Tue Sep 10 13:48:35 UTC 2013 [13:48:49] RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Tue Sep 10 13:48:41 UTC 2013 [13:48:49] RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Tue Sep 10 13:48:41 UTC 2013 [13:48:49] RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Tue Sep 10 13:48:41 UTC 2013 [13:48:49] RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Tue Sep 10 13:48:46 UTC 2013 [13:48:49] RECOVERY - Puppet freshness on cp1065 is OK: puppet ran at Tue Sep 10 13:48:46 UTC 2013 [13:48:50] RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Tue Sep 10 13:48:46 UTC 2013 [13:48:50] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Tue Sep 10 13:48:46 UTC 2013 [13:48:51] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Tue Sep 10 13:48:46 UTC 2013 [13:48:59] RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Tue Sep 10 13:48:51 UTC 2013 [13:48:59] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Tue Sep 10 13:48:51 UTC 2013 [13:48:59] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Tue Sep 10 13:48:51 UTC 2013 [13:48:59] RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Tue Sep 10 13:48:56 UTC 2013 [13:48:59] RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Tue Sep 10 13:48:56 UTC 2013 [13:49:09] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Tue Sep 10 13:49:01 UTC 2013 [13:49:09] RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Tue Sep 10 13:49:01 UTC 2013 [13:49:09] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Tue Sep 10 13:49:06 UTC 2013 [13:49:09] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Tue Sep 10 13:49:06 UTC 2013 [13:49:09] RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Tue Sep 10 13:49:06 UTC 2013 [13:49:39] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Tue Sep 10 13:49:31 UTC 2013 [13:49:39] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Tue Sep 10 13:49:31 UTC 2013 [13:49:39] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Tue Sep 10 13:49:31 UTC 2013 [13:49:39] RECOVERY - Puppet freshness on mw134 is OK: puppet ran at Tue Sep 10 13:49:31 UTC 2013 [13:49:39] RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Tue Sep 10 13:49:31 UTC 2013 [13:49:40] RECOVERY - Puppet freshness on db55 is OK: puppet ran at Tue Sep 10 13:49:36 UTC 2013 [13:49:40] RECOVERY - Puppet freshness on ssl1007 is OK: puppet ran at Tue Sep 10 13:49:36 UTC 2013 [13:49:49] RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Tue Sep 10 13:49:41 UTC 2013 [13:49:49] RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Tue Sep 10 13:49:41 UTC 2013 [13:49:49] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Tue Sep 10 13:49:46 UTC 2013 [13:49:59] RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Tue Sep 10 13:49:51 UTC 2013 [13:49:59] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Tue Sep 10 13:49:51 UTC 2013 [13:49:59] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Tue Sep 10 13:49:51 UTC 2013 [13:50:09] RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Tue Sep 10 13:50:01 UTC 2013 [13:50:09] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Tue Sep 10 13:50:06 UTC 2013 [13:50:09] RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Tue Sep 10 13:50:06 UTC 2013 [13:50:09] RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Tue Sep 10 13:50:06 UTC 2013 [13:50:09] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Tue Sep 10 13:50:06 UTC 2013 [13:50:29] RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Tue Sep 10 13:50:26 UTC 2013 [13:50:29] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Tue Sep 10 13:50:26 UTC 2013 [13:50:29] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Tue Sep 10 13:50:26 UTC 2013 [13:50:39] RECOVERY - Puppet freshness on brewster is OK: puppet ran at Tue Sep 10 13:50:31 UTC 2013 [13:50:49] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:49] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:49] RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:49] RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:49] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:50] RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:50] RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:51] RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:51] RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:52] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:52] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:53] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:53] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Tue Sep 10 13:50:41 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Tue Sep 10 13:50:46 UTC 2013 [13:50:54] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Tue Sep 10 13:50:46 UTC 2013 [13:50:59] RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Tue Sep 10 13:50:51 UTC 2013 [13:50:59] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Tue Sep 10 13:50:51 UTC 2013 [13:50:59] RECOVERY - Puppet freshness on db31 is OK: puppet ran at Tue Sep 10 13:50:56 UTC 2013 [13:50:59] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Tue Sep 10 13:50:56 UTC 2013 [13:50:59] RECOVERY - Puppet freshness on professor is OK: puppet ran at Tue Sep 10 13:50:56 UTC 2013 [13:51:09] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Tue Sep 10 13:51:01 UTC 2013 [13:51:09] RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Tue Sep 10 13:51:01 UTC 2013 [13:51:09] RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Tue Sep 10 13:51:01 UTC 2013 [13:51:09] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Tue Sep 10 13:51:01 UTC 2013 [13:51:09] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:10] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:10] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:11] RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:11] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:12] RECOVERY - Puppet freshness on ssl1006 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:12] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:13] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Tue Sep 10 13:51:06 UTC 2013 [13:51:19] RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Tue Sep 10 13:51:11 UTC 2013 [13:52:06] lovely icinga-wm [13:52:37] RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Tue Sep 10 13:51:52 UTC 2013 [13:52:37] RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Tue Sep 10 13:51:57 UTC 2013 [13:52:37] RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Tue Sep 10 13:51:57 UTC 2013 [13:52:37] RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Tue Sep 10 13:51:57 UTC 2013 [13:52:37] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Tue Sep 10 13:51:57 UTC 2013 [13:56:20] RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Tue Sep 10 13:52:58 UTC 2013 [13:56:20] RECOVERY - Puppet freshness on erzurumi is OK: puppet ran at Tue Sep 10 13:56:18 UTC 2013 [13:56:42] RECOVERY - Puppet freshness on linne is OK: puppet ran at Tue Sep 10 13:56:23 UTC 2013 [13:56:42] RECOVERY - Puppet freshness on db56 is OK: puppet ran at Tue Sep 10 13:56:41 UTC 2013 [13:56:42] RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Tue Sep 10 13:56:41 UTC 2013 [13:56:42] RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Tue Sep 10 13:56:41 UTC 2013 [13:56:42] RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Tue Sep 10 13:56:41 UTC 2013 [14:00:16] RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Tue Sep 10 13:56:47 UTC 2013 [14:00:17] RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Tue Sep 10 13:57:47 UTC 2013 [14:00:17] RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Tue Sep 10 13:58:04 UTC 2013 [14:00:17] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Tue Sep 10 13:58:08 UTC 2013 [14:00:17] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Tue Sep 10 13:58:10 UTC 2013 [14:00:17] RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Tue Sep 10 13:58:13 UTC 2013 [14:02:21] RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Tue Sep 10 14:01:15 UTC 2013 [14:02:21] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Tue Sep 10 14:01:15 UTC 2013 [14:02:21] RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Tue Sep 10 14:01:15 UTC 2013 [14:02:21] RECOVERY - Puppet freshness on formey is OK: puppet ran at Tue Sep 10 14:01:15 UTC 2013 [14:02:21] RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Tue Sep 10 14:01:15 UTC 2013 [14:03:31] RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Tue Sep 10 14:03:21 UTC 2013 [14:03:31] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Tue Sep 10 14:03:23 UTC 2013 [14:03:31] RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Tue Sep 10 14:03:23 UTC 2013 [14:03:33] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Tue Sep 10 14:03:28 UTC 2013 [14:03:33] RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Tue Sep 10 14:03:28 UTC 2013 [14:03:33] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Tue Sep 10 14:03:28 UTC 2013 [14:03:33] RECOVERY - Puppet freshness on search28 is OK: puppet ran at Tue Sep 10 14:03:28 UTC 2013 [14:03:33] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Tue Sep 10 14:03:28 UTC 2013 [14:03:33] RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Tue Sep 10 14:03:28 UTC 2013 [14:03:41] RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Tue Sep 10 14:03:33 UTC 2013 [14:03:41] RECOVERY - Puppet freshness on eeden is OK: puppet ran at Tue Sep 10 14:03:33 UTC 2013 [14:03:42] RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Tue Sep 10 14:03:38 UTC 2013 [14:03:42] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Tue Sep 10 14:03:38 UTC 2013 [14:03:51] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Tue Sep 10 14:03:49 UTC 2013 [14:03:51] RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Tue Sep 10 14:03:49 UTC 2013 [14:03:51] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Tue Sep 10 14:03:49 UTC 2013 [14:03:51] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Tue Sep 10 14:03:49 UTC 2013 [14:03:51] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Tue Sep 10 14:03:49 UTC 2013 [14:04:01] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Tue Sep 10 14:03:54 UTC 2013 [14:04:01] RECOVERY - Puppet freshness on db52 is OK: puppet ran at Tue Sep 10 14:03:54 UTC 2013 [14:04:01] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Tue Sep 10 14:03:59 UTC 2013 [14:04:01] RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Tue Sep 10 14:03:59 UTC 2013 [14:04:11] RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Tue Sep 10 14:04:04 UTC 2013 [14:04:11] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Tue Sep 10 14:04:04 UTC 2013 [14:04:22] RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Tue Sep 10 14:04:14 UTC 2013 [14:04:22] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Tue Sep 10 14:04:14 UTC 2013 [14:05:01] RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Tue Sep 10 14:04:59 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:11] RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:12] RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:12] RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:13] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:13] RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:14] RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:14] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:15] RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:15] RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:16] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:16] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:17] RECOVERY - Puppet freshness on stafford is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:17] RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:18] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:18] RECOVERY - Puppet freshness on es1 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:19] RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:19] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Tue Sep 10 14:05:09 UTC 2013 [14:05:21] RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:21] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:21] RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:21] RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:22] RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:22] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:23] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:23] RECOVERY - Puppet freshness on lanthanum is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:24] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:24] RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:25] RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:25] RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Tue Sep 10 14:05:14 UTC 2013 [14:05:26] RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Tue Sep 10 14:05:19 UTC 2013 [14:05:26] RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Tue Sep 10 14:05:19 UTC 2013 [14:05:27] RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Tue Sep 10 14:05:19 UTC 2013 [14:05:27] RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Tue Sep 10 14:05:19 UTC 2013 [14:05:31] RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Tue Sep 10 14:05:24 UTC 2013 [14:05:31] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Tue Sep 10 14:05:24 UTC 2013 [14:05:31] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Tue Sep 10 14:05:25 UTC 2013 [14:05:31] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Tue Sep 10 14:05:25 UTC 2013 [14:05:32] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Tue Sep 10 14:05:30 UTC 2013 [14:05:32] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Tue Sep 10 14:05:30 UTC 2013 [14:05:41] RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:42] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:42] RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:42] RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:42] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:42] RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:42] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Sep 10 14:05:35 UTC 2013 [14:05:43] RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Tue Sep 10 14:05:40 UTC 2013 [14:05:43] RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Tue Sep 10 14:05:40 UTC 2013 [14:05:51] RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Tue Sep 10 14:05:45 UTC 2013 [14:05:51] RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Tue Sep 10 14:05:50 UTC 2013 [14:05:51] RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Tue Sep 10 14:05:50 UTC 2013 [14:06:01] RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Tue Sep 10 14:05:55 UTC 2013 [14:06:01] RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Tue Sep 10 14:05:55 UTC 2013 [14:06:11] RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Tue Sep 10 14:06:05 UTC 2013 [14:06:11] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Tue Sep 10 14:06:05 UTC 2013 [14:06:11] RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Tue Sep 10 14:06:05 UTC 2013 [14:06:11] RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Tue Sep 10 14:06:10 UTC 2013 [14:06:23] .. [14:06:51] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Tue Sep 10 14:06:41 UTC 2013 [14:07:15] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Tue Sep 10 14:07:01 UTC 2013 [14:07:15] RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Tue Sep 10 14:07:06 UTC 2013 [14:07:15] RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Tue Sep 10 14:07:06 UTC 2013 [14:07:15] RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Tue Sep 10 14:07:06 UTC 2013 [14:07:15] RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Tue Sep 10 14:07:06 UTC 2013 [14:07:48] and stay out :-P [14:08:51] RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:51] RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:51] RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:51] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:51] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:52] RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:52] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Tue Sep 10 14:08:42 UTC 2013 [14:08:53] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:53] RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:54] RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:54] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:55] RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:55] RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:56] RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:56] RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Tue Sep 10 14:08:43 UTC 2013 [14:08:57] RECOVERY - Puppet freshness on search36 is OK: puppet ran at Tue Sep 10 14:08:48 UTC 2013 [14:09:01] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Tue Sep 10 14:08:58 UTC 2013 [14:09:02] RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Tue Sep 10 14:08:58 UTC 2013 [14:09:02] RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Tue Sep 10 14:08:58 UTC 2013 [14:09:02] RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Tue Sep 10 14:08:58 UTC 2013 [14:09:02] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Tue Sep 10 14:08:58 UTC 2013 [14:09:03] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Tue Sep 10 14:08:58 UTC 2013 [14:09:11] RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Tue Sep 10 14:09:03 UTC 2013 [14:09:32] RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:32] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:32] RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:32] RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:32] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:32] RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:33] RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:33] RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:34] RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:34] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Tue Sep 10 14:09:23 UTC 2013 [14:09:35] RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Tue Sep 10 14:09:28 UTC 2013 [14:09:35] RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Tue Sep 10 14:09:28 UTC 2013 [14:09:36] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Tue Sep 10 14:09:28 UTC 2013 [14:09:36] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Tue Sep 10 14:09:28 UTC 2013 [14:09:43] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Tue Sep 10 14:09:38 UTC 2013 [14:09:43] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Tue Sep 10 14:09:38 UTC 2013 [14:09:51] RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Tue Sep 10 14:09:48 UTC 2013 [14:09:52] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Tue Sep 10 14:09:48 UTC 2013 [14:09:52] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Tue Sep 10 14:09:48 UTC 2013 [14:09:52] RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Tue Sep 10 14:09:48 UTC 2013 [14:10:02] RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Tue Sep 10 14:09:53 UTC 2013 [14:10:02] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Tue Sep 10 14:09:53 UTC 2013 [14:10:02] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Tue Sep 10 14:09:58 UTC 2013 [14:10:06] apergos: that only works if you are holding a double barrell with some straw sticking out of your mouth [14:10:11] RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Tue Sep 10 14:10:03 UTC 2013 [14:10:12] RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Tue Sep 10 14:10:04 UTC 2013 [14:10:12] RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Tue Sep 10 14:10:04 UTC 2013 [14:10:21] RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Tue Sep 10 14:10:19 UTC 2013 [14:10:22] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Tue Sep 10 14:10:19 UTC 2013 [14:10:22] RECOVERY - Puppet freshness on db58 is OK: puppet ran at Tue Sep 10 14:10:19 UTC 2013 [14:10:22] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Tue Sep 10 14:10:19 UTC 2013 [14:10:22] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Tue Sep 10 14:10:19 UTC 2013 [14:10:22] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Tue Sep 10 14:10:19 UTC 2013 [14:10:32] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Tue Sep 10 14:10:24 UTC 2013 [14:10:32] RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Tue Sep 10 14:10:24 UTC 2013 [14:10:42] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:43] RECOVERY - Puppet freshness on mw1089 is OK: puppet ran at Tue Sep 10 14:10:34 UTC 2013 [14:10:44] RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Tue Sep 10 14:10:39 UTC 2013 [14:10:51] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Tue Sep 10 14:10:49 UTC 2013 [14:10:52] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Tue Sep 10 14:10:49 UTC 2013 [14:10:52] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Tue Sep 10 14:10:49 UTC 2013 [14:10:52] RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Tue Sep 10 14:10:49 UTC 2013 [14:11:01] RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Tue Sep 10 14:10:54 UTC 2013 [14:11:01] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Tue Sep 10 14:10:54 UTC 2013 [14:11:02] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Tue Sep 10 14:10:54 UTC 2013 [14:11:02] RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Tue Sep 10 14:10:54 UTC 2013 [14:11:02] RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Tue Sep 10 14:10:59 UTC 2013 [14:11:02] RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Tue Sep 10 14:10:59 UTC 2013 [14:11:02] RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Tue Sep 10 14:10:59 UTC 2013 [14:11:03] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Tue Sep 10 14:10:59 UTC 2013 [14:11:03] RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Tue Sep 10 14:10:59 UTC 2013 [14:11:11] RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Tue Sep 10 14:11:04 UTC 2013 [14:11:12] RECOVERY - Puppet freshness on cp1002 is OK: puppet ran at Tue Sep 10 14:11:06 UTC 2013 [14:11:12] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Tue Sep 10 14:11:06 UTC 2013 [14:11:12] RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Tue Sep 10 14:11:06 UTC 2013 [14:11:12] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Tue Sep 10 14:11:06 UTC 2013 [14:11:31] RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:32] RECOVERY - Puppet freshness on rdb1002 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:32] RECOVERY - Puppet freshness on wtp1015 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:32] RECOVERY - Puppet freshness on db34 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:32] RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:32] RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:32] RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:33] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:33] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:34] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Tue Sep 10 14:11:21 UTC 2013 [14:11:34] RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:35] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:35] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:36] RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:36] RECOVERY - Puppet freshness on testsearch1003 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:37] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:37] RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:38] RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:38] RECOVERY - Puppet freshness on db1001 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:39] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:39] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Tue Sep 10 14:11:26 UTC 2013 [14:11:41] RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Tue Sep 10 14:11:31 UTC 2013 [14:11:42] RECOVERY - Puppet freshness on helium is OK: puppet ran at Tue Sep 10 14:11:31 UTC 2013 [14:11:42] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Tue Sep 10 14:11:31 UTC 2013 [14:11:42] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Tue Sep 10 14:11:36 UTC 2013 [14:11:51] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Tue Sep 10 14:11:43 UTC 2013 [14:11:52] RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Tue Sep 10 14:11:48 UTC 2013 [14:11:52] RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Tue Sep 10 14:11:48 UTC 2013 [14:11:52] RECOVERY - Puppet freshness on search18 is OK: puppet ran at Tue Sep 10 14:11:48 UTC 2013 [14:11:52] RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Tue Sep 10 14:11:48 UTC 2013 [14:11:52] RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Tue Sep 10 14:11:48 UTC 2013 [14:12:01] RECOVERY - Puppet freshness on cp1005 is OK: puppet ran at Tue Sep 10 14:11:53 UTC 2013 [14:12:01] RECOVERY - Puppet freshness on sq58 is OK: puppet ran at Tue Sep 10 14:11:53 UTC 2013 [14:12:02] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Tue Sep 10 14:11:53 UTC 2013 [14:12:02] RECOVERY - Puppet freshness on sq54 is OK: puppet ran at Tue Sep 10 14:11:53 UTC 2013 [14:12:02] RECOVERY - Puppet freshness on db1044 is OK: puppet ran at Tue Sep 10 14:11:53 UTC 2013 [14:12:02] RECOVERY - Puppet freshness on mw1032 is OK: puppet ran at Tue Sep 10 14:11:53 UTC 2013 [14:12:02] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Tue Sep 10 14:11:58 UTC 2013 [14:12:11] RECOVERY - Puppet freshness on mc1007 is OK: puppet ran at Tue Sep 10 14:12:03 UTC 2013 [14:12:32] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Tue Sep 10 14:12:29 UTC 2013 [14:12:33] RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Tue Sep 10 14:12:29 UTC 2013 [14:12:33] RECOVERY - Puppet freshness on db63 is OK: puppet ran at Tue Sep 10 14:12:29 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:45] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:46] RECOVERY - Puppet freshness on search33 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:46] RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:47] RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:47] RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Tue Sep 10 14:12:34 UTC 2013 [14:12:48] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Tue Sep 10 14:12:39 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:52] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Tue Sep 10 14:12:44 UTC 2013 [14:12:53] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Tue Sep 10 14:12:49 UTC 2013 [14:13:02] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Tue Sep 10 14:12:54 UTC 2013 [14:13:03] RECOVERY - Puppet freshness on mw1173 is OK: puppet ran at Tue Sep 10 14:12:54 UTC 2013 [14:13:03] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Tue Sep 10 14:12:54 UTC 2013 [14:13:11] RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Tue Sep 10 14:13:04 UTC 2013 [14:13:43] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Tue Sep 10 14:13:39 UTC 2013 [14:13:43] RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Tue Sep 10 14:13:39 UTC 2013 [14:13:43] RECOVERY - Puppet freshness on db48 is OK: puppet ran at Tue Sep 10 14:13:39 UTC 2013 [14:13:43] RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Tue Sep 10 14:13:39 UTC 2013 [14:13:43] RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Tue Sep 10 14:13:39 UTC 2013 [14:13:43] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Tue Sep 10 14:13:39 UTC 2013 [14:14:11] RECOVERY - Puppet freshness on tin is OK: puppet ran at Tue Sep 10 14:14:02 UTC 2013 [14:14:12] RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Tue Sep 10 14:14:02 UTC 2013 [14:14:43] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Tue Sep 10 14:14:33 UTC 2013 [14:14:51] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Tue Sep 10 14:14:45 UTC 2013 [14:15:42] RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Tue Sep 10 14:15:35 UTC 2013 [14:16:02] RECOVERY - Puppet freshness on neon is OK: puppet ran at Tue Sep 10 14:15:55 UTC 2013 [14:16:43] RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Tue Sep 10 14:16:39 UTC 2013 [14:16:43] RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Tue Sep 10 14:16:39 UTC 2013 [14:17:12] RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Tue Sep 10 14:17:05 UTC 2013 [14:18:11] RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Tue Sep 10 14:18:09 UTC 2013 [14:19:30] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Tue Sep 10 14:19:23 UTC 2013 [14:26:06] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Tue Sep 10 14:26:05 UTC 2013 [14:27:06] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Tue Sep 10 14:27:01 UTC 2013 [14:27:16] RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Tue Sep 10 14:27:06 UTC 2013 [14:29:06] RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Tue Sep 10 14:28:59 UTC 2013 [14:29:26] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Tue Sep 10 14:29:20 UTC 2013 [14:29:37] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Tue Sep 10 14:29:31 UTC 2013 [14:29:37] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Tue Sep 10 14:29:31 UTC 2013 [14:30:47] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:48] RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:48] RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:48] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:48] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:48] RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:48] RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:49] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:49] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:50] RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:50] RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Tue Sep 10 14:30:39 UTC 2013 [14:30:51] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Tue Sep 10 14:30:44 UTC 2013 [14:30:51] RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Tue Sep 10 14:30:44 UTC 2013 [14:30:52] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Tue Sep 10 14:30:44 UTC 2013 [14:30:52] RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Tue Sep 10 14:30:44 UTC 2013 [14:30:58] RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Tue Sep 10 14:30:49 UTC 2013 [14:30:58] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Tue Sep 10 14:30:49 UTC 2013 [14:30:58] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Tue Sep 10 14:30:49 UTC 2013 [14:30:58] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Tue Sep 10 14:30:54 UTC 2013 [14:31:07] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Tue Sep 10 14:30:59 UTC 2013 [14:31:08] RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Tue Sep 10 14:30:59 UTC 2013 [14:31:08] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Tue Sep 10 14:31:04 UTC 2013 [14:31:08] RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Tue Sep 10 14:31:04 UTC 2013 [14:31:08] RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Tue Sep 10 14:31:04 UTC 2013 [14:31:08] RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Tue Sep 10 14:31:04 UTC 2013 [14:31:16] RECOVERY - Puppet freshness on db47 is OK: puppet ran at Tue Sep 10 14:31:09 UTC 2013 [14:31:57] RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Tue Sep 10 14:31:54 UTC 2013 [14:31:57] RECOVERY - Puppet freshness on db35 is OK: puppet ran at Tue Sep 10 14:31:54 UTC 2013 [14:31:57] RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Tue Sep 10 14:31:55 UTC 2013 [14:31:57] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Tue Sep 10 14:31:55 UTC 2013 [14:31:57] RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Tue Sep 10 14:31:55 UTC 2013 [14:31:57] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Tue Sep 10 14:31:55 UTC 2013 [14:31:58] RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Tue Sep 10 14:31:55 UTC 2013 [14:32:26] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Tue Sep 10 14:32:25 UTC 2013 [14:39:47] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Tue Sep 10 14:39:39 UTC 2013 [14:40:16] RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Tue Sep 10 14:40:15 UTC 2013 [14:41:02] had a feelin' [14:41:06] RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Tue Sep 10 14:40:58 UTC 2013 [14:47:15] apergos: had a feeling about what? :P [14:47:55] what p858snake said [14:47:59] PROBLEM - search indices - check lucene status page on search1019 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 59205 bytes in 0.015 second response time [14:47:59] PROBLEM - Disk space on cp1047 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 8718 MB (2% inode=99%): /srv/sdb3 8745 MB (2% inode=99%): [14:48:00] PROBLEM - Disk space on ms-be1011 is CRITICAL: DISK CRITICAL - /var/lib/ceph/osd/ceph-121 is not accessible: Input/output error [14:48:08] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [14:48:09] PROBLEM - Disk space on cp1046 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 9787 MB (3% inode=99%): /srv/sdb3 9971 MB (3% inode=99%): [14:54:13] etherpad.wikimedia.org gives invalid certificate warning [15:04:40] RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Tue Sep 10 15:04:38 UTC 2013 [16:13:33] down for everyone or just me? https://ganglia.wikimedia.org/latest/ [16:14:44] (03PS1) 10Aude: (bug 53993) Add ckbwiki prepend sort order and update Wikibase data types [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83637 [16:17:29] bd808: also for me, just said in -tech [16:17:37] if you open specific things it works [16:19:08] wonky [16:33:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.002 second response time [16:40:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.538 second response time [16:45:53] !log reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' [16:45:56] Logged the message, Master [16:47:57] (03PS1) 10Reedy: Update interwiki cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83656 [16:48:45] (03CR) 10Reedy: [C: 032] Update interwiki cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83656 (owner: 10Reedy) [16:49:00] (03Merged) 10jenkins-bot: Update interwiki cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83656 (owner: 10Reedy) [16:51:49] [Tue Sep 10 16:48:54 2013] [error] [client 77.49.50.33] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 113 bytes) in /srv/org/wikimedia/ganglia-web-3.5.7/header.php on line 390 lots of this on nickel [16:56:06] PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 1 unauthenticated, 0 locked, 0 copy to table, 38 statistics [16:57:15] RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 12 statistics [16:57:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [17:00:11] ok well I upped the memory limit in /etc/php5/apache2/php.ini to 192MB from 128MB (live hack, not in puppet) but sure would like to know why it suddenly needs more than it used to [17:05:00] !log ganglia was failign to serve some urls, complaining that memory size of 134217728 bytes exhausted, I upped to 192mb temporarily in /etc/php5/apache2/php.ini, needs followup [17:05:04] Logged the message, Master [17:09:16] PROBLEM - Disk space on professor is CRITICAL: DISK CRITICAL - free space: /a 21691 MB (3% inode=99%): [17:13:58] PROBLEM - Parsoid on wtp1008 is CRITICAL: Connection refused [17:13:58] PROBLEM - Disk space on cp1058 is CRITICAL: DISK CRITICAL - free space: /srv/sda3 3623 MB (1% inode=99%): /srv/sdb3 3615 MB (1% inode=99%): [17:13:58] PROBLEM - search indices - check lucene status page on search1017 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60624 bytes in 0.044 second response time [17:13:58] PROBLEM - Backend Squid HTTP on sq42 is CRITICAL: Connection refused [17:13:58] PROBLEM - search indices - check lucene status page on search16 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 58689 bytes in 0.113 second response time [17:14:07] PROBLEM - RAID on db1001 is CRITICAL: CRITICAL: Degraded [17:20:04] !log rebooting sq42 [17:20:08] Logged the message, Master [17:21:41] PROBLEM - Host sq42 is DOWN: PING CRITICAL - Packet loss = 100% [17:27:44] (03PS1) 10Reedy: Initial disable of ContributionReporting/Contributiontracking [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83664 [17:31:51] PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 40 statistics [17:32:53] RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 2 statistics [17:48:24] (03PS1) 10Reedy: Remove ContributionReporting and ContributionTracking [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83668 [17:52:20] !log removing sq42 from pybal [17:52:23] Logged the message, Master [18:09:09] !log reedy synchronized php-1.22wmf14/extensions/WikibaseDataModel [18:09:11] Logged the message, Master [18:09:41] !log reedy synchronized php-1.22wmf14/extensions/Wikibase [18:09:44] Logged the message, Master [18:10:05] Well, isn't that just a no-op [18:10:49] !log reedy synchronized php-1.22wmf15/extensions/WikibaseDataModel [18:10:52] Logged the message, Master [18:11:18] !log reedy synchronized php-1.22wmf15/extensions/Wikibase [18:11:21] Logged the message, Master [18:20:04] (03PS1) 10Reedy: Kill wmf9 stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83672 [18:20:49] (03CR) 10Reedy: [C: 032] Kill wmf9 stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83672 (owner: 10Reedy) [18:20:59] (03Merged) 10jenkins-bot: Kill wmf9 stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83672 (owner: 10Reedy) [18:28:36] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.119 second response time [18:28:37] (03CR) 10Mwalker: [C: 04-1] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83664 (owner: 10Reedy) [18:28:45] (03CR) 10Mwalker: [C: 04-1] "(3 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83668 (owner: 10Reedy) [18:33:42] (03PS2) 10Reedy: Initial disable of ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83664 [18:35:15] idk if this is normal or related to problems above. but... go to a file from just about any commit and click the HEAD link. gives 500 [18:35:25] https://git.wikimedia.org/blob/operations%2Fpuppet.git/5f04abefc65dc872b0503afe9e95e521ad9e0941/manifests%2Fmisc%2Finstall-server.pp links to [18:35:30] https://git.wikimedia.org/blob/operations%2Fpuppet.git/HEAD/manifests%2Fmisc%2Finstall-server.pp which is the 500 [18:36:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.863 second response time [18:42:11] (03PS3) 10Reedy: Initial disable of ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83664 [18:42:18] (03CR) 10Reedy: [C: 032] Initial disable of ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83664 (owner: 10Reedy) [18:42:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:34] (03Merged) 10jenkins-bot: Initial disable of ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83664 (owner: 10Reedy) [18:43:20] !log reedy synchronized wmf-config/InitialiseSettings.php 'Disable ContributionReporting' [18:43:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.557 second response time [18:43:24] Logged the message, Master [18:47:10] (03Abandoned) 10Reedy: Remove ContributionReporting and ContributionTracking [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83668 (owner: 10Reedy) [18:48:39] (03PS1) 10Jeremyb: commonswiki wgImportSources, add some sources [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83678 [18:48:47] (03CR) 10jenkins-bot: [V: 04-1] commonswiki wgImportSources, add some sources [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83678 (owner: 10Jeremyb) [18:50:03] (03PS1) 10Reedy: Remove ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83680 [18:52:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [18:54:50] (03PS2) 10Jeremyb: commonswiki wgImportSources, add some sources [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83678 [18:57:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [19:00:05] PROBLEM - Varnish HTTP bits on cp3019 is CRITICAL: Connection timed out [19:00:54] RECOVERY - Varnish HTTP bits on cp3019 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.870 second response time [19:13:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [19:27:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.603 second response time [19:40:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.846 second response time [20:06:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.168 second response time [20:13:32] (03PS1) 10Dzahn: fix typo, wikifur is .com not .org [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/83755 [20:15:30] (03CR) 10Dzahn: [C: 032] fix typo, wikifur is .com not .org [operations/debs/wikistats] - 10https://gerrit.wikimedia.org/r/83755 (owner: 10Dzahn) [20:17:57] (03CR) 10Dzahn: [C: 032] "thanks, this works better" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83348 (owner: 10Jeroen De Dauw) [20:19:39] (03CR) 10Dzahn: [C: 032] Maintenance for http://fr.planet.wikimedia.org/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/83620 (owner: 10Dereckson) [20:23:13] PROBLEM - DPKG on db1045 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [20:24:33] apergos: FYI, (a) feel free to ping my cell in case of Labs breakage at any time (b) start-nfs is a script that does all the magic. :-) [20:24:45] greg-g: Still ok to do a quick deploy? [20:25:00] Coren: yeah I heard you hopped on some hours back and made it all better [20:25:12] RECOVERY - DPKG on db1045 is OK: All packages OK [20:25:22] so do you have time/interest to give me the 1 minute 'here's how it all fits together' [20:25:28] so I can do it right the next time? [20:25:29] apergos: I'm going to add myself to the paging list and make sure I got some icinga alerts for this too though. [20:25:49] I mean I looked at the script but I'm still not 100% [20:26:11] there's the manage volumes piece too [20:26:28] apergos: Yeah, the reason NFS doesn't autostart is because of the cold swap setup between labstore3 and labstore4; the raid array on the shielves isn't fired up automatically because we don't want it running on both at the same time. [20:26:42] logical [20:26:53] Yeah, manage volumes is politely restarted by start-nfs; that's the bit that adds new volumes when projects are added. [20:26:59] ok so to stop it in an orderly way, I would...? [20:27:14] ah, I was whacking at that by hand as I didn't know better [20:27:20] but without /srv not much was happening :-D [20:27:33] Right, /srv gets made live by start-nfs [20:27:39] gotcha [20:27:57] so is there a nice termination sequence too? or not so much? [20:28:27] Shutdown actually cleanly stops everything; when you want to be extra paranoid, you can stop the kernel-nfs-server which will make the filesystems idle; but it's not strictly required. [20:28:39] (03CR) 10Dzahn: [C: 032] Make defaults for StatsD module more sensible [operations/puppet] - 10https://gerrit.wikimedia.org/r/83602 (owner: 10Ori.livneh) [20:28:56] (It's mostly useful if you want to unmount them to do maintenance; if you're only going to reboot it's not critical) [20:29:20] ok, I did that but I'm not sure which order I did those in :-D [20:29:35] remembering last time the reboot that turned into ou having to copy stuff from snapshots [20:29:41] This should probably be all documented, but I've been spending so much time trying to track down the hardware issue docs always get pushed to "later". :-( [20:29:43] I was determined to avoid that at all costs [20:30:07] erm yeah, even a quick paragraph 'do this to restart, do this afer reboot' would be great [20:30:22] apergos: That didn't so much happen because of the reboot than it did because the controller failed before the reboot. By the time we did the unmount, it was already too late and the block device had corrupted. [20:30:23] I was this close to uncommenting those lines in the fstab [20:31:09] ok well that makes me feel better about my stab in the dark then :-D [20:31:32] Yeah, if you had it wouldn't have been a catastrophe; just really confused everyone. You'd have brought back the pre-corruption filesystem. :-) [20:31:38] :-D [20:31:46] ok that's also good to know! [20:32:37] But as a rule, never hesitate to make my phone noisy. I'm going to try to find some of the actual work time we have left over the next two weeks to document labstore[34] [20:34:22] RECOVERY - Disk space on analytics1011 is OK: DISK OK [20:42:20] !log csteipp synchronized php-1.22wmf16/extensions/CentralAuth [20:42:23] Logged the message, Master [20:42:40] !log csteipp synchronized php-1.22wmf15/extensions/CentralAuth [20:42:43] Logged the message, Master [20:45:55] (03CR) 10Dzahn: [C: 032] jobs/careers redirects on wikimedia and wikipedia [operations/apache-config] - 10https://gerrit.wikimedia.org/r/83134 (owner: 10Dzahn) [20:49:08] csteipp: yessir [20:49:12] glad you did it [20:49:13] :) [20:49:24] (03CR) 10Siebrand: "> Yes, I saw. Would you be annoyed if I fixed that in core?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/79231 (owner: 10MaxSem) [20:54:04] what's BC? [20:54:08] re last grrrit-wm link above [20:56:54] !log demon synchronized php-1.22wmf16/extensions/CirrusSearch 'Logging for annoying parser output missing bug' [20:56:57] Logged the message, Master [21:03:07] !log csteipp synchronized php-1.22wmf15/extensions/CentralAuth [21:16:36] (03PS1) 10Dzahn: make those jobs/career redirects [R=301,L,NE] [operations/apache-config] - 10https://gerrit.wikimedia.org/r/83758 [21:16:41] (03CR) 10jenkins-bot: [V: 04-1] make those jobs/career redirects [R=301,L,NE] [operations/apache-config] - 10https://gerrit.wikimedia.org/r/83758 (owner: 10Dzahn) [21:18:03] (03PS2) 10Dzahn: make those jobs/career redirects [R=301,L,NE] [operations/apache-config] - 10https://gerrit.wikimedia.org/r/83758 [21:18:47] (03CR) 10Dzahn: [C: 032] "dzahn@fenari:~$ apache-fast-test jobs-careers.url mw1044" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/83758 (owner: 10Dzahn) [21:23:23] (03PS1) 10Chad: Use Cirrus as default on mw.org [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83759 [21:26:17] !log LocalisationUpdate completed (1.22wmf16) at Tue Sep 10 21:26:17 UTC 2013 [21:26:21] Logged the message, Master [21:31:24] (03PS3) 10Chad: Remove obsolete sudo setup [operations/puppet] - 10https://gerrit.wikimedia.org/r/82763 [21:31:44] <^d> Ryan_Lane: Trivial ^ [21:33:50] !log temp. hacked a fork limit into apache-graceful-all so my local ssh agent doesn't die when it's too fast, bleh:p [21:33:53] Logged the message, Master [21:34:00] :-D [21:34:32] (03CR) 10Ryan Lane: [C: 032] Remove obsolete sudo setup [operations/puppet] - 10https://gerrit.wikimedia.org/r/82763 (owner: 10Chad) [21:39:03] (03PS1) 10Aaron Schulz: Removed ceph backend config for decommissioning [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83761 [21:39:37] (03CR) 10Aaron Schulz: "Shed no tears :|" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83761 (owner: 10Aaron Schulz) [21:40:13] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:13] awww [21:41:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.381 second response time [21:42:01] (03PS1) 10Dereckson: Maintenance for http://fr.planet.wikimedia.org/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/83762 [21:44:35] if [ `cat /etc/cluster` == pmtpa ]; then [21:45:01] /home/wikipedia/bin/apache-graceful-all: line 6: syntax error near unexpected token `then' .. [21:45:32] apergos: sigh, but it's done anyways :) [21:45:54] aww [21:48:19] what happens if ther eis no /etc/cluster? maybe you want to assign that to a var and put double quotes around it in the if [21:50:22] (03PS1) 10Cmjohnson: decomming sq42 --another one bites the dust [operations/puppet] - 10https://gerrit.wikimedia.org/r/83764 [21:50:24] !log reedy synchronized php-1.22wmf16/includes/search/SearchEngine.php [21:50:29] Logged the message, Master [21:50:32] !log LocalisationUpdate completed (1.22wmf15) at Tue Sep 10 21:50:32 UTC 2013 [21:50:35] Logged the message, Master [21:50:48] yay [21:51:38] 997 13010 0.0 0.0 12496 1732 pts/10 S+ 21:04 0:00 /bin/bash /usr/local/bin/l10nupdate-1 [21:51:39] 997 14430 0.0 0.0 4400 612 ? Ss 02:00 0:00 /bin/sh -c /usr/local/bin/l10nupdate-1 >> /var/log/l10nupdatelog/l10nupdate.log 2>&1 [21:51:39] 997 14431 0.0 0.0 12436 1684 ? S 02:00 0:00 /bin/bash /usr/local/bin/l10nupdate-1 [21:51:40] 997?> [21:51:46] (03CR) 10Cmjohnson: [C: 032] decomming sq42 --another one bites the dust [operations/puppet] - 10https://gerrit.wikimedia.org/r/83764 (owner: 10Cmjohnson) [21:53:04] Reedy: I was wondering that. uid of l10nupdate? [21:53:15] Presumably [21:53:23] uh oh [21:53:50] l10nupdate@tin:/a/common$ id [21:53:50] uid=997(l10nupdate) gid=10002(l10nupdate) groups=10002(l10nupdate) [22:02:56] (03PS2) 10Dereckson: Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 (owner: 10TTO) [22:03:11] (03CR) 10Dereckson: [C: 031] Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 (owner: 10TTO) [22:03:20] PROBLEM - DPKG on db1045 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:03:44] (03CR) 10jenkins-bot: [V: 04-1] Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 (owner: 10TTO) [22:04:20] !log reedy synchronized php-1.22wmf16/extensions/Wikibase [22:04:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:23] Logged the message, Master [22:04:38] (03CR) 10Faidon Liambotis: [C: 04-1] "professor is currently lucid so while I agree fully on the idea, this won't work :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83603 (owner: 10Ori.livneh) [22:05:24] (03PS1) 10Dereckson: Enable Extension:NewUserMessage on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83765 [22:06:24] (03PS2) 10Reedy: (bug 53993) Add ckbwiki prepend sort order and update Wikibase data types [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83637 (owner: 10Aude) [22:06:29] (03CR) 10Reedy: [C: 032] (bug 53993) Add ckbwiki prepend sort order and update Wikibase data types [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83637 (owner: 10Aude) [22:07:00] (03Merged) 10jenkins-bot: (bug 53993) Add ckbwiki prepend sort order and update Wikibase data types [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83637 (owner: 10Aude) [22:07:53] !log reedy synchronized wmf-config/ [22:07:56] Logged the message, Master [22:08:59] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 10 22:08:58 UTC 2013 [22:09:03] Logged the message, Master [22:12:19] !log dist-upgrade and reboot boron [22:12:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.949 second response time [22:12:22] Logged the message, Master [22:13:20] RECOVERY - DPKG on db1045 is OK: All packages OK [22:15:43] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikivoyage and wikidatawiki to 1.22wmf16 [22:15:47] Logged the message, Master [22:16:03] (03PS1) 10Reedy: Wikivoyage and wikidatawiki to 1.22wmf16 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83766 [22:17:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:19:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.539 second response time [22:21:51] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: No successful Puppet run in the last 10 hours [22:22:23] (03CR) 10Reedy: [C: 032] Wikivoyage and wikidatawiki to 1.22wmf16 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83766 (owner: 10Reedy) [22:22:26] ^d: Hi. I want to try and help with hhvm. I noticed hhtm.wmflabs is down. Was wondering if you can guide me on what needs to be done in that area (robla referred me) [22:22:33] (03Merged) 10jenkins-bot: Wikivoyage and wikidatawiki to 1.22wmf16 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83766 (owner: 10Reedy) [22:22:59] (03PS2) 10Reedy: Remove ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83680 [22:23:04] (03CR) 10Reedy: [C: 032] Remove ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83680 (owner: 10Reedy) [22:23:15] (03Merged) 10jenkins-bot: Remove ContributionReporting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83680 (owner: 10Reedy) [22:24:16] !log reedy synchronized wmf-config/ [22:24:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:50] yhager: ^d hasn't got a laptop atm [22:25:58] hasn't got/isn't at a [22:26:10] (03PS4) 10Ryan Lane: WORK IN PROGRESS: Simplify git-deploy configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/83046 [22:26:19] Reedy: okay, I'll try and catch him later then. thx. [22:27:22] (03PS5) 10Ryan Lane: WORK IN PROGRESS: Simplify git-deploy configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/83046 [22:27:28] You can of course test locally [22:30:00] Reedy: the question is what is still missing to make it work for wikimedia. the basic mediawiki works, I think some of the extensions maybe aren't. [22:31:15] Reedy: I have it running locally. [22:31:33] We need things packaged/debian-ised for deployment too [22:32:43] Reedy: hhvm? [22:32:51] Yeah [22:33:33] I believe fb has debian/ubuntu packages. [22:34:10] https://github.com/facebook/hiphop-php/wiki/Prebuilt-Packages-on-Debian-7-%28wheezy%29 [22:35:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.659 second response time [22:38:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:39:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.559 second response time [22:50:55] (03PS1) 10Dzahn: add salt grains for applicationserver roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/83768 [22:52:32] (03PS3) 10TTO: Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 [22:58:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.878 second response time [23:05:28] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 301 seconds [23:06:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:48] (03CR) 10Dzahn: [C: 031] "http://www.projecthoneypot.org/ip_218.248.46.167" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 (owner: 10TTO) [23:09:28] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 301 seconds [23:11:04] (03CR) 10Dzahn: [C: 032] Maintenance for http://fr.planet.wikimedia.org/ [operations/puppet] - 10https://gerrit.wikimedia.org/r/83762 (owner: 10Dereckson) [23:11:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.323 second response time [23:13:05] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 305 seconds [23:13:42] (03PS2) 10Dereckson: Enable Extension:NewUserMessage on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83765 [23:22:08] (03CR) 10Ori.livneh: "Yep. Does another host come to mind?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83603 (owner: 10Ori.livneh) [23:28:15] !log starting mariadb 10 multi-source replication test from all clusters on db1045 [23:28:18] Logged the message, Master [23:28:18] (03Abandoned) 10Dzahn: redirect http->https on etherpad.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/80314 (owner: 10Dzahn) [23:29:57] (03CR) 10Dzahn: [C: 032] Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 (owner: 10TTO) [23:30:09] (03Merged) 10jenkins-bot: Add temporary IP throttle exception for Kannada workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83613 (owner: 10TTO) [23:30:22] does it blend? eh.. merge:) [23:32:16] !log dzahn synchronized ./wmf-config/throttle.php [23:32:19] Logged the message, Master [23:33:23] !log dzahn synchronized ./wmf-config/throttle.php [23:42:57] icinga's borked, checking it out now [23:43:00] oh noes [23:43:03] neon is borked ? [23:44:10] doh, i would !log this but .... [23:44:12] rebooting neon [23:46:16] !log rebooted neon, had crashed [23:46:19] Logged the message, Mistress of the network gear. [23:47:51] again? [23:48:01] huh. i can't ping en.wikipedia.org. [23:48:02] it died earlier, well last night sometime when I was asleep [23:48:08] oh really ? [23:48:10] hrm :-/ [23:48:10] yes [23:48:17] Pinging wikipedia-lb.esams.wikimedia.org [91.198.174.225] with 32 bytes of data: [23:48:21] Request timed out. [23:48:25] I didn't notice til much later in my day, rebooted, it was ok [23:48:27] it is broken or just me? [23:48:28] seemingly [23:48:30] MatmaRex: traceroute? [23:48:48] I can ping it from SF [23:48:55] (using the lb hostname) [23:49:11] Reedy: running… [23:51:05] meh, probably my provider or something. http://pastebin.com/q8cKp2YW [23:52:05] anyway it seems to work now. :)