[00:01:09] those IPs translate to: vanadium, tin, fenari and neon [00:01:47] yeah, that's in site.pp (grep for tcpircbot) [00:01:55] i'm not sure why it's filtering connections from those hosts now [00:02:07] i blamed faidon's change above but i think i may have been too hasty [00:02:30] could it hit the max_clients for some reason? [00:02:34] i see a 5 there [00:03:38] mutante: could be; try 'service tcpircbot-logmsgbot restart' [00:05:02] ori-l: oh, the user it's running as doesn't look that good [00:05:12] 996 [00:05:42] !log restarted logmsgbot [00:05:51] Logged the message, Master [00:05:53] yeah, didn't work [00:06:18] you can test by running: dologmsg "\!log ori testing dologmsg" [00:06:19] on tin [00:06:50] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [00:07:40] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 00:07:36 UTC 2013 [00:07:50] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [00:09:22] tcpircbot:x:996:1002 [00:10:06] there's something listening on that address, it's just closing the connection instantly [00:10:30] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 00:10:21 UTC 2013 [00:10:50] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [00:10:53] which is consistent with the incoming ip not matching the whitelisted CIDRs [00:11:45] oh, it logs to stderr [00:11:50] which upstart should be capturing [00:12:25] what are the last few lines in /var/log/upstart/tcpircbot-logmsgbot.log ? [00:12:39] (thanks for helping, btw) [00:12:48] 2013-07-02 00:04:45,923 NickServ!NickServ@services. ['You are now identified for \x02logmsgbot\x02.'] [00:13:05] i just see how it connects to the ircd [00:13:13] any 'Connection from's? [00:13:41] ah, yes [00:13:47] from IPv6 addresses [00:13:57] just from v6 [00:14:00] can you paste one, so i can test it? [00:14:19] i guess the kernel upgrade did that [00:14:30] but it should work with ipv6 too, so must be a bug [00:15:12] actually, " IPv4-mapped IPv6 " [00:15:27] 2013-07-01 10:07:13,409 Connection from ('::ffff:10.64.0.196', 59233, 0, 0) [00:16:06] starting with ::ffff: is for those special ones [00:16:52] and yea, it's tin [00:18:41] heh [00:18:41] https://gerrit.wikimedia.org/r/#/c/69625/1/manifests/site.pp [00:18:58] New patchset: Ori.livneh; "Revert "tcpircbot: IPv4 cidr instead of IPv4-mapped IPv6"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71566 [00:19:18] ah, and reverted [00:20:19] New review: Dzahn; "root@neon:~# grep -i connect /var/log/upstart/tcpircbot-logmsgbot.log" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71566 [00:22:39] New patchset: Dzahn; "Revert "tcpircbot: IPv4 cidr instead of IPv4-mapped IPv6"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71566 [00:24:29] New review: Manybubbles; "Almost done." [operations/debs/jmxtrans] (debian) C: -1; - https://gerrit.wikimedia.org/r/71079 [00:24:37] New review: Dzahn; "Connection from ('::ffff:10.64.0.196' matches tin and is what is trying to connect in /var/log/upsta..." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71566 [00:24:38] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71566 [00:25:18] a puppet run on neon should refresh the service [00:25:22] yep [00:26:20] hrmm nagios/icinga runs take a while [00:27:33] nope [00:27:37] is it running with the updated config? [00:28:05] no, that puppet run was already started before the merge it seems [00:28:42] watches puppet.log for stuff to happen [00:29:13] but it must have pulled the change otherwise it would have not restarted the service [00:30:31] nah, it's just been sitting at the current puppet run since before merge [00:30:36] will check it in a couple minutes [00:30:53] i did that getting impatient [00:31:41] heh :) [00:52:16] mutante: still running? [00:53:31] - "cidr": "10.64.21.123/32,10.64.0.196/32,208.80.152.165/32,127.0.0.1/32", [00:53:34] + "cidr": "::ffff:10.64.21.123/128,::ffff:10.64.0.196/128,::ffff:208.80.152.165/128,::ffff:127.0.0.1/128", [00:53:37] happened right this second [00:53:43] and yeah, puppet run ongoing [00:53:44] i felt it! [00:53:52] :D [00:54:48] \!log mutante fix logmsgbot? [00:54:48] \!log ori testing dologmsg. mutante ftw. [00:54:50] heh [00:54:53] :) [00:54:57] the \ was not required [00:55:00] but .. the restart was manual [00:55:08] puppet didnt do it [00:55:10] !log ori testing dologmsg. mutante ftw. [00:55:19] Logged the message, Master [00:55:41] \!log mutante fix logmsgbot? [00:55:53] nice [00:56:49] thanks mutante:) [00:56:56] yw! [00:56:56] * ori-l is so done with irc bots [00:57:02] oh, heh, yea [00:57:12] thanks for looking at them [00:57:59] mutante: thank you for fixing me; I feel much better now [00:58:19] * ori-l pets logmsgbot [00:58:31] :) [00:58:32] on that note, home [00:58:38] cya [01:01:36] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.001299738884 secs [01:02:16] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:03] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:09] New patchset: Manybubbles; "Initial Debian packaging for jmxtrans." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [01:10:05] New review: Manybubbles; "I believe this covers my latest complaint and Faidon's." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [01:21:34] !log updated Parsoid to 65bd7ae9 [01:21:45] Logged the message, Master [01:33:33] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.001928925514 secs [01:36:37] New patchset: Ottomata; "Puppetizing hive client, server and metastore." [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/71569 [01:43:05] New patchset: Ottomata; "Updating README.md" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/71570 [01:43:23] Change merged: Ottomata; [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/71570 [01:55:01] New review: Ottomata; "Ok, I just updated the README, and cleaned up some issues on Github." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71248 [02:07:16] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [02:09:03] !log LocalisationUpdate completed (1.22wmf8) at Tue Jul 2 02:09:03 UTC 2013 [02:09:13] Logged the message, Master [02:15:57] !log LocalisationUpdate completed (1.22wmf9) at Tue Jul 2 02:15:57 UTC 2013 [02:16:07] Logged the message, Master [02:19:41] Hi morebots. [02:22:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:13] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 2 02:23:13 UTC 2013 [02:23:22] Logged the message, Master [02:23:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [02:29:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [02:39:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.837 second response time [02:47:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [02:56:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.150 second response time [03:05:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [03:10:12] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [03:23:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.145 second response time [03:27:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.145 second response time [03:31:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.406 second response time [03:57:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [04:01:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [04:06:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:09:24] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 04:09:19 UTC 2013 [04:10:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:10:24] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 04:10:19 UTC 2013 [04:11:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:12:14] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 04:12:04 UTC 2013 [04:12:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:12:44] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 04:12:43 UTC 2013 [04:13:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:13:24] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 04:13:17 UTC 2013 [04:14:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:23:17] New patchset: Tim Starling; "let systemuser l10nupdate have bash as shell so it can run l10nupdate scripts via dsh on new mw* hosts." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71525 [04:23:27] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71525 [04:28:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [05:05:50] !log testing LU with a real shell (not /bin/false) [05:05:59] Logged the message, Master [05:06:50] why on earth does it need its own user account? what's wrong with mwdeploy? [05:07:56] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [05:08:21] mwdeploy can't log in [05:08:39] l10nupdate is essentially equivalent to an administrative user, it can sync things [05:09:26] mwdeploy is relatively unprivileged [05:10:01] both are badly named, then [05:10:03] I think it's very scary giving l10nupdate the level of permissions it has, but at least it's only that one script [05:10:22] ah, i see [05:10:28] well, maybe it's not misnamed, then [05:13:36] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:14:00] !log LocalisationUpdate completed (1.22wmf8) at Tue Jul 2 05:14:00 UTC 2013 [05:14:08] Logged the message, Master [05:15:06] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [05:15:56] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [05:15:56] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:56] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:56] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:56] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:56] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [05:15:56] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:57] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:57] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:48] !log LocalisationUpdate completed (1.22wmf9) at Tue Jul 2 05:21:48 UTC 2013 [05:21:57] Logged the message, Master [05:23:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:52] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 2 05:25:52 UTC 2013 [05:26:01] Logged the message, Master [05:28:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.151 second response time [05:40:06] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:40:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:06] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:41:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.149 second response time [06:07:05] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:28:35] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 06:28:30 UTC 2013 [06:29:05] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:29:15] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 06:29:09 UTC 2013 [06:30:05] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:31:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:57:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.014 second response time [07:01:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [07:10:04] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [07:11:06] New patchset: Tim Starling; "Keep memcached memory out of swap" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23872 [07:11:40] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23872 [07:13:15] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [07:17:10] New patchset: Faidon; "Kill media-storage thumbs server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71583 [07:17:14] apergos: ^ [07:17:30] TimStarling: oh heh, that's one of the oldest I had in my outgoing queue [07:17:38] TimStarling: thanks! [07:20:13] you can kill the stuff in site.pp for sure; the labs media server setup will need something akin to the thumbs classes you are removing, though I don't plan to use those directly, so, meh, your call about those [07:20:53] which setup? [07:21:05] the beta labs media server setup [07:21:36] what about it? [07:21:48] are there any plans about that? I have no idea :) [07:23:07] it's running but needs to be puppetized, we don't have a miniature ceph of swift cluster in there but an instance that serves original or scaled media, forwarding requests to image scalers as needed for nonexistent thumbs [07:23:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.173 second response time [07:24:43] the purging stuff I imagine we will never use (since no multicast there), and like I say I expect to redo the thumb classes but.. dunno, should we just toss em? anyways like I say your call [07:25:14] well as I say in the commit message they're going to be in git history forever [07:25:26] but I don't want people to try to modularize that as is, that'd be a waste of time [07:25:34] or hold other transitions, e.g. nginx or whatever [07:27:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.919 second response time [07:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [07:44:27] New patchset: Faidon; "Kill media-storage thumbs server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71583 [07:45:48] New review: Faidon; "Indeed. Init.d uses --make-pidfile (so start-stop-daemon creates it), so having rsyncd write the pid..." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71105 [07:45:49] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71105 [07:46:59] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71583 [07:48:14] New patchset: Faidon; "Convert swift's rsyncd to the new rsync module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71107 [07:51:26] apergos: btw, are you still working on h310->h710 for swift? [07:51:33] just wondering :) [07:51:39] I've kinda lost touch [07:53:05] I shall be; I'd been making sure all the hosts are at 100% (they are) and all the disks that were out of the rings are back in and working (need to double check that again but think it's done) [07:54:04] cool [07:54:42] good to hear some good news considering how cursed media storage is [07:55:02] between c2100s, h310s, swift & ceph bugs [07:55:32] or gluster and nfs if you want even more storage problems [07:56:26] heh yes [07:56:36] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71107 [07:57:02] TimStarling: merging on sockpuppet for you [07:57:16] TimStarling: (memcached -k) [08:02:29] New review: Faidon; "The old config doesn't have uid/gid, so it runs as root. The new config defaults to nobody, so I'm g..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/71106 [08:04:12] New patchset: Faidon; "rsync: there is no group nobody, it's nogroup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71588 [08:04:53] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71588 [08:09:53] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [08:10:12] !log nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master' [08:10:22] Logged the message, Master [08:12:16] New patchset: Nikerabbit; "ULS deployment phase 4" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71319 [08:12:35] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71319 [08:15:37] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS on enwiki part 1' [08:15:46] Logged the message, Master [08:16:33] !log nikerabbit synchronized wmf-config/CommonSettings.php 'ULS on enwiki part 2' [08:16:43] Logged the message, Master [08:19:29] apergos: I pushed the new rsyncd confs to all swift nodes, let's see if it still works :) [08:21:57] uh oh :-D [08:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:32:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:36] !log nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master' [08:32:45] Logged the message, Master [08:33:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [08:33:33] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.001404523849 secs [08:35:43] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:38:16] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [08:41:07] chrismcmahon: hi [08:41:17] chrismcmahon: you are the one responsible for dev access? [08:41:47] Kelson: bonjour :) [08:41:57] hashar: salut! [08:42:01] chrismcmahon is most probably sleeping at this time, he is in middle us [08:42:05] te faut quoi ? :) [08:42:26] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:42:49] hashar: un peu perdu dans le système de gestion des accès. Je suis normalement responsable pour le code rep. openzim et je veux donner accès à un autre dev. (tntnet) [08:43:08] hashar: mais je sais pas trop comment faire - ni même si j'ai vraiment suffisement de permissions [08:43:34] in private to avoid cluttering this channel with french :-D [08:44:16] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [08:45:51] !log gerrit : added Tntnet user to the 'openzim' group. Grants him push/submit rights on the openzim git repository. Requested by Kelson. [08:46:00] Logged the message, Master [08:52:29] New patchset: Faidon; "Initial Debian packaging for jmxtrans" [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [08:53:30] gerrit... [08:53:31] so slow... [08:53:41] gerrit or jenkins jobs reporting ? :D [08:53:51] gerrit [08:53:55] "Working..." [08:53:57] oh there are no jobs for those repositories yet :D [08:54:16] we talked a bit about jmxtrans yesterday, probably going to have it installed for Jenkins as well [08:54:25] cool! [08:54:47] hey hashar, can we move zuul.pp and misc/contint.pp out of manifests/ ? [08:54:58] into modules or role classes [08:55:32] the manifests/zuul.pp , that is the ugly class zuulwikimedia [08:55:40] merely a wrapper around the zuul module [08:55:50] but yeah it could probably get merged in the role::zuul [08:55:56] I am going to upgrade Jenkins, would take roughly half an hour to complete. [08:56:46] (I was the one that put the new version into apt btw :) [08:57:30] so yeah I'm a bit worried someone else from ops will try to move zuul and all that as part of the modularization effort [08:57:36] but you know much better what needs to happen [08:58:40] let me break^Wupgrade jenkins first then we can talk about the zuul.pp [08:58:46] thank you for the jenkins upload ! [08:58:56] yep, sure, no hurry [09:00:06] !log gallium : upgraded Jenkins from 1.509.1 to 1.509.2 {{bug|50522}} [09:00:14] Logged the message, Master [09:00:55] New patchset: Mark Bergsma; "Switch cp3007 and cp3008 frontends to chash weight 100" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71590 [09:02:03] mark: jenkins restarting so there will be no puppet lint for you :D [09:02:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71590 [09:02:58] LOST [09:04:13] New review: Faidon; "Fixed a few minor things here and there." [operations/debs/jmxtrans] (debian) C: -1; - https://gerrit.wikimedia.org/r/71079 [09:06:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:10] mark: beta no more have any squid instance \O/ installing the text varnish took me like 3 minutes [09:07:20] good [09:07:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [09:07:34] it probably helps that I take beta a little bit into account now too ;) [09:07:39] I need to adjust the HTCP multicast routing system in MediaWiki [09:07:55] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [09:08:01] the purges can only be sent to one address/port [09:08:52] yeah [09:09:11] could make it accept a list of addresses [09:09:37] the format is roughly: array( 'url regex' => array( 'host', 'port' ) ) [09:09:45] that second array could be an array of hosts/ports [09:10:02] yes [09:10:17] in prod, I guess you will want several multicast groups ? [09:10:26] yes [09:13:14] I might even write some tests :D [09:14:49] !log reinstalling dataset1001 [09:14:59] Logged the message, Master [09:17:05] PROBLEM - Host dataset1001 is DOWN: CRITICAL - Host Unreachable (208.80.154.11) [09:22:15] RECOVERY - Host dataset1001 is UP: PING OK - Packet loss = 0%, RTA = 0.86 ms [09:24:15] PROBLEM - NFS on dataset1001 is CRITICAL: Connection refused [09:24:25] PROBLEM - SSH on dataset1001 is CRITICAL: Connection refused [09:24:35] PROBLEM - Lighttpd HTTP on dataset1001 is CRITICAL: Connection refused [09:25:59] !log jenkins upgraded. [09:26:07] Logged the message, Master [09:27:02] apergos: one of the puppet todo items modularizing lighttpd or replacing it altogether [09:27:19] apergos: my preference would be the latter, nginx seems like a good fit to me [09:27:27] apergos: so you might want to convert dataset configs into nginx :) [09:28:10] will consider it later [09:28:24] (busy now) [09:29:40] nah, just saying [09:29:44] since I saw the lighttpd alert [09:31:25] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.001474738121 secs [09:32:05] k [09:33:53] New review: Nemo bis; "Just a note that the current patch is still an incomplete draft, per last review by MZMcBride." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/69982 [09:36:17] PROBLEM - NTP on dataset1001 is CRITICAL: NTP CRITICAL: No response from NTP server [09:40:20] New patchset: Nemo bis; "[WIP] Enable CAPTCHA for all edits of non-confirmed users on pt.wikipedia in order to reduce editing activity" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/69982 [09:45:27] RECOVERY - SSH on dataset1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [09:49:17] RECOVERY - NFS on dataset1001 is OK: TCP OK - 0.001 second response time on port 2049 [09:49:37] RECOVERY - Lighttpd HTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1684 bytes in 0.007 second response time [09:50:50] New patchset: MaxSem; "Alternative way of setting resource paths" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71381 [09:52:47] PROBLEM - Host dataset1001 is DOWN: CRITICAL - Host Unreachable (208.80.154.11) [09:56:39] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71381 [10:00:18] !log maxsem synchronized wmf-config/mobile.php [10:00:28] Logged the message, Master [10:00:55] New patchset: MaxSem; "Revert "Alternative way of setting resource paths"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71592 [10:01:23] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71592 [10:02:45] New patchset: MaxSem; "Revert "Revert "Alternative way of setting resource paths""" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71593 [10:02:46] !log maxsem synchronized wmf-config/mobile.php [10:06:24] New patchset: MaxSem; "Alternative way of setting resource paths" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71593 [10:07:14] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [10:07:43] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71593 [10:09:20] !log maxsem synchronized wmf-config/mobile.php [10:14:00] New patchset: MaxSem; "Fix double wmfSetupMobileLoadScript() invocation" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71594 [10:15:06] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71594 [10:17:00] !log maxsem synchronized wmf-config/mobile.php [10:57:29] mark: I have sent the multiple hosts HTCP routing patch for mediawiki :) Want me to add you as a reviewer ? [11:03:02] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [11:08:49] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [11:09:59] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [11:15:04] sure! [11:17:37] New patchset: Mark Bergsma; "Set all esams upload frontends to chash weight 100" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71599 [11:20:37] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71599 [11:22:11] New review: Faidon; "puppet is an ops tool, as are a few others; puppet is the wrong tool for this, I'm not sure why we n..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62404 [11:34:29] New patchset: Mark Bergsma; "Put all old varnish cache servers (temporarily) in decommission list" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71601 [11:35:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71601 [11:48:01] New review: Mark Bergsma; "(1 comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/71380 [11:48:41] New patchset: Mark Bergsma; "Give Gabriel shell on the Parsoid Varnishes again" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71535 [11:49:28] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71535 [11:56:18] New review: Mark Bergsma; "For all the reasons already mentioned above..." [operations/puppet] (production) C: -2; - https://gerrit.wikimedia.org/r/62404 [11:57:26] New review: QChris; "I think so, as github's master cannot be fast-forwarded to gerrit's" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71248 [11:58:31] New patchset: Mark Bergsma; "role::cache get rid of unrecognized escape sequences" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71329 [11:59:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71329 [12:01:41] New patchset: Siebrand; "Enable EducationProgram on Dutch language Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71605 [12:02:39] New review: Siebrand; "Needs to wait for community approval on bug 50574." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/71605 [12:09:38] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [12:09:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:38] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [12:09:48] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:48] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:58] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:58] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:58] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:08] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:18] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [12:10:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [12:10:28] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:28] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:28] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:38] RECOVERY - Host dataset1001 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [12:14:48] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:16:44] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:20:32] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:21:09] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:21:28] PROBLEM - Host dataset1001 is DOWN: CRITICAL - Host Unreachable (208.80.154.11) [12:22:45] RECOVERY - Host dataset1001 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [12:23:49] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:28:41] have people seen this behavior from puppet where it writes out /etc/network/interfaces for link aggregation with the bonded interface first, eth0/1 etc afterwards, and the network doesn't come up? (Just had this on precise and changing the order fixed it) [12:31:19] paravoid: thanks for deploying it [12:32:43] akosiaris: I am reading your puppet testing snippet. Would it be safe to run untrusted code with 'puppet apply --noop' ? [12:33:08] hopefully --noop does nothing :-] [12:33:25] hashar: that is the idea [12:33:44] what do you mean untrusted though? what's the context ? [12:34:10] akosiaris: the idea would be to have Jenkins run the puppet tests whenever someone submit a change in Gerrit against ops/puppet [12:34:30] akosiaris: so potentially an attacker could inject some weird code in a manifest and have it realized as user jenkins-slave on the gallium server :) [12:34:58] well i have never seen --noop actually realize something [12:35:01] to workaround that , we only run linting tests for submitted patches. Actual code running is limited to whitelisted people (aka wikimedia / wikidata + few trusted volunteers) [12:35:08] yeah I guess it is safe [12:35:49] χμμμ wait... there is a chance... [12:35:55] let me check something [12:37:22] anyway most of the people interacting with ops/puppet are whitelisted so it does not really matter [12:42:02] hashar: so.. unsafe [12:42:29] mostly due to the fact that modules can define puppet parser functions and facts [12:42:56] thank you for the check! :-] [12:43:01] these are arbitrary ruby code nested under the lib/ directory of each puppet modules [12:43:04] module* [12:43:47] you are welcome. Sad about the outcome though. I 'd love the tests in jenkins :-( [12:44:52] akosiaris: there were talks about sandboxing all that [12:45:16] talks by who ? [12:45:24] people :) [12:45:24] New review: Manybubbles; "I'll fix the symlink." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [12:45:28] em included [12:45:32] manybubbles|away: hey [12:45:37] the rough idea is to have jenkins tests isolated in disposable sandboxes [12:45:44] I'm here! [12:45:45] we thought about using vagrant for that [12:46:18] manybubbles: didn't mean to push it back to you, but I guessed it'd be easy for you [12:46:51] paravoid: I've got like 10 years of experience hacking build.xmls so it shouldn't be bad [12:47:11] paravoid: actually, while I've got you, where should the symlink point? to where the files will eventually be installed? [12:47:22] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:48:46] what's the point of the symlink? :) [12:48:58] that's /usr/share/jmxtrans/log, what is that needed for? [12:49:11] should it be a symlink to /var/log/jmxtrans/ ? [12:49:17] or can jmxtrans just write there? [12:52:03] apergos: beta thumbnails are broken :D https://bugzilla.wikimedia.org/show_bug.cgi?id=50499 [12:52:21] apergos: turns out deployment-upload attempt to access de.wikipedia.beta.wmflabs.org which is not reachable because of nat [12:56:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71607 [12:57:27] paravoid: in fact jmxtrans writes there directly. I'm not 100% sure why the jmxtrans authors wanted the symlink, but it was one of the things they originally went to great pains to build. My question was more around whether there is debian magic that shifts the symlink or whether it needs to be absolute. [13:00:46] they were not broken, it used to access deployment-squid [13:00:57] has something changed over there recently? hashar [13:01:22] apergos: yup, I got rid of the squid text cache in favor of a varnish one [13:01:43] so deployment-squid will no longer work [13:01:57] indeed [13:02:28] and I forgot about the thumbnails [13:03:59] ariel@deployment-upload:/data/project/upload7/scripts$ grep squid * [13:03:59] thumb-handler.php:$reqURL = "http://deployment-squid/w/thumb.php?"; [13:04:05] that's what you'll need to fix [13:04:15] just edit in place [13:04:38] okk [13:04:51] I saw the report yesterday but had other things broken on my plate, so... [13:05:08] I was grepping by IP :-) [13:05:30] ah ha [13:05:53] bha http://upload.beta.wmflabs.org/wikipedia/de/thumb/6/6a/VLine.svg/500px-VLine.svg.png :D [13:05:56] some german message yeahhh [13:06:59] errror generating thumbnail (says google) [13:07:37] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:37] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:37] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:37] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:37] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:37] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [13:07:57] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:57] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:57] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:57] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:07] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:07] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [13:08:07] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [13:08:17] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:17] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:17] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:17] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:28] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [13:12:00] New patchset: Manybubbles; "Initial Debian packaging for jmxtrans." [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [13:18:40] New review: Akosiaris; "LGTM" [operations/debs/buck] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/70673 [13:18:41] Change merged: Akosiaris; [operations/debs/buck] (master) - https://gerrit.wikimedia.org/r/70673 [13:24:32] mkdir: cannot create directory `/sys/fs/cgroup/memory/mediawiki/job/23391': No such file or directory [13:24:33] limit.sh: failed to create the cgroup. [13:24:34] Anyone familiar with cgroups ? =) [13:26:09] am I the only one still having trouble with gerrit? [13:26:49] paravoid: what kind of troubles ? [13:26:58] Nikerabbit complained earlier about being logged out from time to time [13:27:04] Are you using Firefox? [13:27:08] I'm getting "Working..." [13:27:09] I am [13:27:16] for Gerrit slowness, ^demon & qchris are you best option [13:27:40] <^demon> We're still trying to track down the firefox thing [13:27:45] * Nemo_bis was unable to open any change on firefox this morning [13:27:58] in the last hour or so, sometimes it works [13:28:51] ^demon: great, thanks. let me know if I help / be your guinea pig or something [13:28:54] Nemo_bis: By �sometimes it works� do you mean �for a given change it sometimes works� or �for some changes it works, for others it doesn't� [13:29:12] New review: Faidon; "I don't know much about build.xml but diff makes sense." [operations/debs/jmxtrans] (wikimedia) C: 1; - https://gerrit.wikimedia.org/r/71384 [13:29:37] The gerrit debug mode should allow you to view the pages (as a temporary workaround. Yes it sucks :-( [13:29:52] You need to add the dbg=1 URL paramater, to use that. [13:30:42] manybubbles: if debian/patches is making things more difficult for you we can revert AzaToth's change [13:30:54] manybubbles: other than that, I think it's ready to go [13:31:07] should I wait for r71384? [13:31:59] paravoid: if you don't mind the clean issue then we don't have to wait [13:32:08] paravoid: uh? [13:32:30] I don't [13:32:34] * AzaToth is totally innocent [13:32:37] AzaToth: debian/patches [13:32:41] AzaToth: for jmxtrans [13:32:41] paravoid: I don't mind the debian/patches thing now that it is done. [13:32:43] paravoid: yes [13:32:50] I replied there [13:33:48] manybubbles: awesome, thanks [13:33:59] * paravoid does another build before the +2 [13:34:11] AzaToth and paravoid: thanks so much for reviewing this. [13:34:24] thanks for working on it [13:34:30] also, should I do the tagging? [13:34:31] manybubbles: haven't reviewed it [13:34:40] and I have to say, very nice work, especially for a first time [13:35:00] AzaToth: you complained about things worth complaining about, which is helpful. [13:35:10] paravoid: thanks [13:35:31] New review: AzaToth; "I would assume it's fine to pre-drop this files" [operations/debs/jmxtrans] (wikimedia) C: 1; - https://gerrit.wikimedia.org/r/71384 [13:37:05] paravoid: sadly format 3.0 (git) hasn't been made stable ヾ [13:37:17] the format is stable, the archive doesn't accept it [13:37:20] and never will if you ask me [13:37:39] prolly [13:37:57] I meant "accepted" [13:38:04] ottomata: heya [13:38:33] hiya [13:38:50] busy? [13:39:40] nope! just starting to get busy [13:39:43] whatsup? [13:41:57] New patchset: Mark Bergsma; "Don't show the error page on HTTP 200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71614 [13:42:26] qchris_away: I've not managed to understand/keep track [13:43:40] ottomata: I made more changes to the rsync module, it's really not meant for Debian [13:43:45] but we should be done by now [13:43:50] the other thing is [13:43:51] https://gerrit.wikimedia.org/r/#/c/53714/5 [13:43:57] hashar abandonded it (bad hashar) [13:44:09] ahh [13:44:12] dawww [13:44:12] feel free to restore it :-) [13:44:15] we were so close! [13:44:16] haha [13:44:33] paravoid, re rsync module, that's cool [13:44:38] New patchset: Mark Bergsma; "Don't show the error page on HTTP 200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71614 [13:44:39] i probably won't have time to work on it soon anyway [13:44:40] I abandoned it because there have been no activity for a couple months and it started being annoying in my dashboard hehe [13:45:13] ok [13:45:21] do you want to finish the geoip module or should I? [13:45:26] I'm going to need it this week, so... :) [13:45:39] i can finish that, i'm not sure what's left, we were just arguing about the .dat files, right? [13:45:48] I don't remember! [13:45:51] hah [13:46:01] so, i think everything was good, except our last two comments [13:46:07] you didn't liek the files:/// .dat files [13:46:10] and I responded with why they were there [13:46:50] right [13:46:55] I stand by my previous comment :-) [13:47:00] hashar: thanks for fixing the thumbnail-bug on beta [13:47:10] your comment was that I should,n't use files:// from a module [13:47:11] but I' ot [13:47:12] i'm not [13:47:13] se4598: that was a nasty issue :/ [13:47:21] it just looks taht way because this is being exracted out of manifests [13:47:28] New review: Manybubbles; "I know this doesn't help a ton with the review but the build.xml change causes the javadoc directory..." [operations/debs/jmxtrans] (wikimedia) - https://gerrit.wikimedia.org/r/71384 [13:47:52] New patchset: Mark Bergsma; "Don't show the error page on HTTP 200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71614 [13:48:49] oh, and I think I should get rid of misc::geoip [13:49:00] that should be role::geoip? [13:49:24] New patchset: Mark Bergsma; "Don't show the error page on HTTP 200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71614 [13:49:30] uhm, that's a poor fit for a role [13:49:34] k [13:49:35] geoip is not a role [13:49:45] just as much as having puppet is not a role [13:50:03] I'd say that we should copy this single call to roles on a case-by-case [13:50:12] not all systems want city accurate data [13:50:17] k [13:50:21] hmm, i guess I do have puppet:///files/GeoIP as a default value for $puppet_source, hmm [13:50:22] hm [13:50:25] yeah that's probably not good [13:50:26] hm [13:50:38] man been a while since I looked at this [13:51:04] ok, the idea with keeping the .dat files in the repo here, was that people could use the puppet $data_provider, for example in labs, and have it still work [13:51:30] buuut, i guess i'm not even using that by default [13:51:47] in misc::geoip, I'm just using the maxmind geoip data package [13:51:49] for labs [13:51:50] by default [13:52:13] paravoid: do you think I should just rip out the .dat files and not set a for $puppet_source [13:52:13] ? [13:52:20] a default* [13:52:34] I think so, yes [13:52:45] ok. [13:52:54] actually I think the whole class might do too much magic [13:53:01] ha awwww [13:53:11] (you might be right…i had reason for the magic of course but awwww) [13:53:14] reasons* [13:53:23] I know and I'm deliberating that myself [13:53:31] but the more I look at it again the more I get confused [13:53:48] but I don't mind much, whatever you say [13:54:01] I'll cope ;) [13:54:03] i'm looking [13:54:38] which magic don't you like? the inherits? [13:54:48] i think that's what I don't like, since parameterized classes with inherits gets confusing [13:54:55] i tested this one real good and it works like this [13:55:10] but it was not easy to understand how defaults and non specified parameters on child classes get set [13:55:28] well, the geoip-database package and the commerical .dats have different filenames [13:55:34] so one is not a replacement for the other [13:55:48] bwwaaaaa wha? [13:55:51] and yet we have the same class, geoip, to accumulate those different use cases [13:56:30] geoip-database provides GeoIP.dat & GeoIPv6.dat, i.e. country-level databases [13:56:45] then there's the geoip-database-contrib package which downloads at runtime GeoLite [13:57:04] am I using database-contrib? [13:57:33] New patchset: Mark Bergsma; "Don't show the error page on HTTP 200" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71614 [13:57:41] which is country + less accurate city/asn [13:57:43] for v4/v6 [13:58:05] then we have the commercial ones [13:58:50] which are GeoIP.dat for the country one [13:58:59] plus GeoIPCity.dat and GeoIPRegion.dat [13:59:23] hm, oooooook, so it sounds like the proper thing to do would be to support multiple .dat providers at once, and have one selected via or symlink / update-alternatives or something [13:59:55] I think we should just have three different classes for different purposes under the same module [14:00:03] instead of data_providers [14:00:44] and explicitly include geoip::commercial or geoip::geolite or something [14:00:46] that would do what I just suggested? [14:00:51] i think that's cool [14:01:01] lrwxrwxrwx 1 root root 31 Jul 2 16:58 GeoIPCity.dat -> /etc/alternatives/GeoIPCity.dat [14:01:05] yeah [14:01:08] that's geoip-database-contrib [14:01:11] oh [14:01:17] heheh, it sounds someone had that idea of yours already [14:01:24] *seems [14:01:53] strangely enough the rest are not alternatives though [14:01:55] so it's kinda useless [14:02:00] -rw-r--r-- 1 root root 3438400 Jun 18 16:07 GeoIPASNum.dat [14:02:00] lrwxrwxrwx 1 root root 31 Jul 2 16:58 GeoIPCity.dat -> /etc/alternatives/GeoIPCity.dat [14:02:04] -rw-r--r-- 1 root root 581110 Jun 5 01:03 GeoIP.dat [14:02:06] -rw-r--r-- 1 root root 1038407 Jun 7 23:29 GeoIPv6.dat [14:02:09] -rw-r--r-- 1 root root 17648217 Jun 5 03:34 GeoLiteCity.dat [14:03:42] so, paravoid, 3 classes, with an $alternative => true parameter? [14:03:55] and a comment saying only use one with alternative => true! [14:04:03] I don't think realistically we'd ever use multiple classes on one box [14:04:05] and then /usr/share/GeoIP directories are symlinks [14:04:08] yeah, probably not [14:04:10] hm [14:04:17] so sure, if you're feeling like it [14:04:20] and if we did that'd get confusing [14:04:20] but I wouldn't spend time on it :) [14:04:24] yeahhh hm [14:04:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71614 [14:04:28] i think you might be right [14:04:50] but my point for separate classes is not coinstability [14:04:59] it's that they provide different things, so it should be explicit [14:05:01] just that they do ifferent hings [14:05:01] yeah [14:05:02] makes sense [14:05:16] hm, ok, that change shouldn't be hard to make [14:05:21] lemme at it, [14:05:23] :) [14:06:00] well, hm i mean, paravoid [14:06:07] I do ahve 3 separate classes that do different things [14:06:13] its just that the interface to them is abstracted [14:06:25] you're just asking me to remove the interface [14:06:43] oh hah [14:06:59] okay, I really need to have a look to remember what that does [14:07:00] or I'm just going to confuse us both [14:07:03] ha, ok [14:07:13] basically, there are 3 geoip::data:: classes [14:07:22] but, you don't have to include them manually [14:07:32] geoip has a data_provider parameter [14:08:05] 'puppet', 'maxmind' and 'package' are the valid values [14:08:11] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [14:08:11] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [14:08:21] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [14:08:21] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:21] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:24] or 'none' or false or whatever [14:08:24] sounds fine? :) [14:08:31] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:31] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:31] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:31] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:31] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:31] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:31] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:32] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:32] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:33] if you set data_provider => false, then it won't install any data [14:08:33] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:34] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [14:08:37] mark: ? [14:08:41] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [14:08:41] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [14:08:50] what ottomata describes sounds fine I mean [14:08:51] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [14:09:01] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [14:09:01] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [14:09:01] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [14:09:21] mark: the three classes or data_provider? [14:09:29] (i'm still waiting for gerrit, grrr) [14:09:44] oh wait, I can just git fetch [14:12:35] okay [14:12:37] sounds fine for now [14:12:43] no -contrib support, but we can add it later [14:12:53] a missing logrotate for ::maxmind too, but we can add that later too [14:13:00] hm ok! ha, i'll remove the .dat file stuff [14:13:24] and whatever you want to do about misc/geoip [14:13:29] oh yeah hm [14:14:09] I'm in a trusting mood [14:14:14] happy with whatever you want :-) [14:14:14] hahah [14:14:45] you're only saying that because I agreed with you and then realized that was already done :p! [14:15:09] haha no [14:15:21] geoip is such a trivial thing, I don't want to bikeshed it much [14:15:29] I'm really trying man [14:15:42] haha [14:21:46] hey mark, paravoid: would you guys have some time to look at Snaps varnishkafka commit at https://gerrit.wikimedia.org/r/#/c/70928/ [14:22:00] New patchset: Jgreen; "manually route donate.wikimedia.org mail to aluminium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71380 [14:22:00] New patchset: Jgreen; "add donate.wikimedia.org to secondary mx relay_domains" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71382 [14:22:44] drdee: bblack already did and I'm guessing mark will also considering he knows varnish inside-out [14:23:05] drdee: so no point with me also looking at it [14:25:59] New review: Jgreen; "spaces to tabs fixed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71382 [14:26:57] re [14:27:05] https://gerrit.wikimedia.org/r/#/c/71382/ [14:28:42] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:29:08] so I got that funny error on labs where /sys/fs/cgroup/memory/mediawiki/job/#### was not existing [14:29:22] that required me to restart the upstart service 'mw-cgroups' which was already running [14:29:32] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [14:29:45] I am wondering whether we could write something in puppet to have it notify the service whenever the /sys/fs/cgroup/memory/mediawiki/ directory does not exist. Thoughts ? [14:32:32] paravoid: fair point [14:33:45] ok, paravoid, i dunno what to do with misc::geoip if it isn't a 'role'. i understand why it isn't a role, but I would like to abstract the differences between production and labs [14:34:01] I don't think someone who wants to use this module in production should have to go and look up the volatile file path [14:34:26] i guess I could move that bit into the module, but I just cringe at accessing $::realm from the module [14:35:33] I think it's okay to move it the 4 lines to the callers [14:35:46] a production system might be okay with geoip-database for example [14:36:05] New review: Andrew Bogott; "The rsync module uses 0/0 as the default values in conf file sections. So I think this is equivalen..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71106 [14:36:07] also consider what we previously said about different filenames/databases [14:36:22] there are 5 callers right now [14:36:48] ja, sure, if someone wanted something special, they don't have to use misc::geoip [14:36:57] the use of the geoip module on puppetmaster is different [14:37:04] because it uses data_provider => maxmind [14:37:25] right [14:39:49] so you think I should move the use of geoip module into each caller to misc::geoip? [14:40:19] duplicating puppet://volatile/GeoIP everywhere? [14:50:24] hashar, can you spare a few minutes before you clock out for the night? [14:50:46] andrewbogott: sure [14:50:59] andrewbogott: congrats on your puppet coding guide :-]  Reaallly useful [14:51:31] It was mostly already-written stuff that I just patched together :) But, glad it's useful. [14:51:42] hashar, I want to apply https://gerrit.wikimedia.org/r/#/c/71106/ and make sure it works... [14:52:20] but I don't feel especially qualified to verify that it's working… would you be able to test that the rsync bits are still working, or tell me how to test that? [14:52:23] New patchset: Ottomata; "Move geoip to a module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53714 [14:52:54] changing locations, back in a bit [14:53:17] andrewbogott: potentially on beta though I am not sure whether we use rsync as well [14:53:20] I need someone to merge and deploy a mw-config change after community decision (AFTv5 for dewiki), https://gerrit.wikimedia.org/r/71546 [14:53:35] se4598: ask in #wikimedia-dev :-) [14:54:01] andrewbogott: ah for search. beta has two search instances which are still running puppetmaster:self [14:54:17] andrewbogott: so we could use them as playground. [14:54:29] hashar: OK -- if I apply to those instances, how will I know if it's broken or not afterwards? [14:55:21] andrewbogott: i think it used to sync the search index from the indexing box to the search boxes. Maybe lucene-jobs.sh let me look [14:56:13] * hashar digs in ./files/lucene/lucene.jobs.sh [14:58:05] !log - update mwlib to 0.15.9 [14:58:14] Logged the message, Master [14:58:37] !log restarted all services [14:58:46] Logged the message, Master [15:01:22] andrewbogott: so that might be inside lucene search java extension deployment-search01.pmtpa.wmflabs shows a bunch of syncing errors. I guess it has a build in rsync clinet [15:01:45] But it's already not working, huh? [15:01:51] hi, can someone copy solr1001:/var/log/jetty/*.stderrout.log* to my home dir on bast1001, please? [15:01:52] no idea [15:02:25] andrewbogott: na it seems to be working fine [15:02:40] ok. If I apply that patch will you be able to tell if it's still working? [15:02:49] /a/search/indexes/update/simplewiki.prefix on deployment-search01.pmtpa.wmflabs shows updates from a few hours ago [15:03:30] oh my god [15:03:32] it is in a cron [15:03:41] or not hmm [15:06:11] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [15:06:21] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:21] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:21] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:21] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:21] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:21] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:22] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:22] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:23] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:23] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:24] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:24] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:31] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [15:06:31] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:31] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:51] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [15:06:51] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:51] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [15:07:01] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [15:07:01] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [15:07:01] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [15:11:19] andrewbogott: so lucene-search on search boxes does shell out to rsync [15:11:28] andrewbogott: I have no idea how to trigger the index refresh though [15:11:44] hashar, how often was it running? [15:12:01] andrewbogott: apparently once a day around 9am UTC. [15:12:18] Oh, that's a long time to wait to see if it still works [15:12:45] restaritng it [15:12:53] New review: Mark Bergsma; "Solid work. :)" [operations/software/varnish/varnishkafka] (master) - https://gerrit.wikimedia.org/r/70928 [15:13:09] Change merged: Faidon; [operations/debs/jmxtrans] (debian) - https://gerrit.wikimedia.org/r/71079 [15:13:38] New patchset: Jgreen; "manually route donate.wikimedia.org mail to aluminium, fixed spaces-to-tabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71380 [15:14:28] !log apt: include jmxtrans 242-1 [15:14:37] Logged the message, Master [15:14:41] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:41] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [15:15:52] andrewbogott: I have no clue sorry :/ notpeter or ^demon might be able to tell how to trigger the lucene search index rsync [15:16:01] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [15:16:01] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:01] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:01] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:01] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:01] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [15:16:02] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:02] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:03] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [15:16:40] hashar, when it runs does it rsync from both index hosts? [15:16:50] there is only one index iirc [15:16:55] New review: Faidon; "This will do for now, thanks a lot." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/53714 [15:16:56] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53714 [15:16:56] deployment-searchidx01.pmtpa.wmflabs [15:17:04] New review: Hashar; "This can be tested on the beta labs instances for search:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71106 [15:17:09] deployment-searchidx02, deployment-searchidx01 [15:17:15] ah the second is not used [15:17:19] I should delete it probably [15:18:00] yeah that is an instance without puppetmaster::self [15:18:53] deletedd :) [15:19:17] andrewbogott: so you could fetch your patch under /var/lib/git/operations/puppet , run puppet and see what happens tomorrow :-) [15:19:24] or have someone figure out how to trigger the rsync [15:19:41] New review: Nemo bis; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71546 [15:20:04] hashar: Yep, my patch is live on the index machine. [15:20:16] thanks mark! [15:22:02] New review: Daniel Kinzler; "@JanZerebecki: that sounds plausible! Daniel, Katie: have you looked into this?" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [15:22:44] New review: Andrew Bogott; "My patch is now live on deployment-searchidx01. We'll see how the next sync does." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71106 [15:22:58] hashar, there's no reason for me to patch deployment-search01 is there/ [15:22:59] ? [15:23:11] probably not :-] [15:23:33] seems the rsync daemon is only on the indexer [15:23:54] and lucene-search conf on the search box is most probably still pointing to its IP + /a/search/index [15:26:18] Looking at the diff in the new conf file, I'm pretty confident that this change is a noop. [15:26:33] But, best to test this once in the real world before I apply this same process to eight other systems :) [15:40:59] New patchset: BBlack; "s/outbuf_size/outbuf_bytes/ to reduce confusion" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71621 [15:40:59] New patchset: BBlack; "Move comment to appropriate place, remove redundant return" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71622 [15:40:59] New patchset: BBlack; "Dynamic HTTP receive buffer size in case of large error mssages" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71623 [15:40:59] New patchset: BBlack; "Indentation fixup (whitespace-only)" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71624 [15:41:00] New patchset: BBlack; "minor tweak for socket reuse" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71625 [15:41:00] New patchset: BBlack; "NEWS / version bump for 0.0.7" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71626 [15:41:48] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71621 [15:42:33] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71622 [15:42:45] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71623 [15:43:20] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71624 [15:43:38] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71625 [15:44:10] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71626 [15:47:46] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71380 [15:53:05] New patchset: BBlack; "Merge branch 'master' into debian" [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71627 [15:53:06] New patchset: BBlack; "bump pkg version" [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71628 [15:53:13] New patchset: Jgreen; "add donate.wikimedia.org to secondary mx relay domains" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71629 [15:53:28] Change merged: BBlack; [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71627 [15:53:41] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71629 [15:53:49] Change merged: BBlack; [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71628 [15:56:20] PROBLEM - spamassassin on sodium is CRITICAL: Connection refused by host [15:56:36] PROBLEM - mailman on sodium is CRITICAL: Connection refused by host [15:59:17] New patchset: Reedy; "Disable AFTv5 feedback submission on dewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71546 [16:01:32] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71546 [16:03:01] !log reedy synchronized wmf-config/InitialiseSettings.php [16:03:10] Logged the message, Master [16:07:18] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:07:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:19] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:19] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:20] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:21] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:21] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:22] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:07:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:07:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:07:48] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:07:42 UTC 2013 [16:07:48] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:07:43 UTC 2013 [16:07:48] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:07:46 UTC 2013 [16:07:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:07:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:07:58] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:07:50 UTC 2013 [16:08:10] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:07:58 UTC 2013 [16:08:10] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:08:14 UTC 2013 [16:08:18] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:08:15 UTC 2013 [16:08:18] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:08:16 UTC 2013 [16:08:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:08:29] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:08:27 UTC 2013 [16:08:30] New patchset: Ottomata; "Installing git-review on stat nodes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71632 [16:08:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:38] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:08:30 UTC 2013 [16:08:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:45] !log authdns update, switching mx records for donate.wikimedia.org to mchenry/sodium [16:08:48] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71632 [16:08:48] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:08:40 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:08:44 UTC 2013 [16:08:49] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:08:47 UTC 2013 [16:08:55] Logged the message, Master [16:09:08] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:08:58 UTC 2013 [16:09:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:09:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:18] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:09:13 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:09:16 UTC 2013 [16:09:18] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:09:16 UTC 2013 [16:09:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:09:28] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:09:23 UTC 2013 [16:09:28] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:33] !log updated vhtcpd package on brewster to 0.0.7-1 [16:09:38] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:09:28 UTC 2013 [16:09:38] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:09:29 UTC 2013 [16:09:38] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:09:30 UTC 2013 [16:09:38] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:09:35 UTC 2013 [16:09:38] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:09:37 UTC 2013 [16:09:42] Logged the message, Master [16:09:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:09:48] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:09:40 UTC 2013 [16:09:58] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:09:48 UTC 2013 [16:09:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:09:58] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:09:49 UTC 2013 [16:09:58] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:09:53 UTC 2013 [16:10:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:09] !log depooling mw80 [16:10:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:18] Logged the message, Master [16:10:18] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:28] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:10:17 UTC 2013 [16:10:28] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:10:17 UTC 2013 [16:10:28] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:10:21 UTC 2013 [16:10:29] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:10:25 UTC 2013 [16:10:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:38] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:10:33 UTC 2013 [16:10:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:49] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:10:46 UTC 2013 [16:10:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:10:58] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:10:50 UTC 2013 [16:10:58] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:10:53 UTC 2013 [16:11:08] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:11:00 UTC 2013 [16:11:08] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:11:03 UTC 2013 [16:11:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:11:14 UTC 2013 [16:11:19] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:11:14 UTC 2013 [16:11:20] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:11:14 UTC 2013 [16:11:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:11:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:11:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:38] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:11:29 UTC 2013 [16:11:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:48] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:11:46 UTC 2013 [16:11:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:11:50 UTC 2013 [16:11:58] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:11:50 UTC 2013 [16:11:58] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:11:53 UTC 2013 [16:11:58] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:11:56 UTC 2013 [16:12:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:08] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:11:59 UTC 2013 [16:12:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:08] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:12:00 UTC 2013 [16:12:08] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:12:00 UTC 2013 [16:12:09] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:09] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:12:04 UTC 2013 [16:12:09] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:12:05 UTC 2013 [16:12:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:19] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:12:09 UTC 2013 [16:12:20] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:12:17 UTC 2013 [16:12:29] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:12:23 UTC 2013 [16:12:47] New patchset: Ottomata; "Fixing geoip include for statistics nodes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71634 [16:12:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:12:48] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:12:43 UTC 2013 [16:12:48] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:12:47 UTC 2013 [16:12:58] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71634 [16:12:58] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:12:49 UTC 2013 [16:12:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:12:52 UTC 2013 [16:13:08] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:12:59 UTC 2013 [16:13:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:18] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:13:18] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:13:14 UTC 2013 [16:13:18] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:13:14 UTC 2013 [16:13:19] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:13:14 UTC 2013 [16:13:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:13:28] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:13:24 UTC 2013 [16:13:38] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:13:27 UTC 2013 [16:13:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:38] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:13:36 UTC 2013 [16:13:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:13:38 UTC 2013 [16:13:58] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:13:54 UTC 2013 [16:13:58] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:13:57 UTC 2013 [16:14:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:14:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:18] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:14:09 UTC 2013 [16:14:18] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:14:12 UTC 2013 [16:14:18] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:14:12 UTC 2013 [16:14:18] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:14:15 UTC 2013 [16:14:19] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:14:15 UTC 2013 [16:14:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:14:28] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:14:21 UTC 2013 [16:14:28] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:14:21 UTC 2013 [16:14:28] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:14:21 UTC 2013 [16:14:29] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:14:25 UTC 2013 [16:14:29] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:14:26 UTC 2013 [16:14:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:38] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:14:37 UTC 2013 [16:14:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:14:48] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:14:45 UTC 2013 [16:14:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:14:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:14:58] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:14:51 UTC 2013 [16:15:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:08] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:15:01 UTC 2013 [16:15:08] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:15:02 UTC 2013 [16:15:08] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:15:07 UTC 2013 [16:15:14] !log rebooting mw80 for bios update [16:15:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:20] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:15:20] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:21] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:15:09 UTC 2013 [16:15:21] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:15:14 UTC 2013 [16:15:23] Logged the message, Master [16:15:38] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:15:27 UTC 2013 [16:15:38] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:15:28 UTC 2013 [16:15:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:38] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:15:30 UTC 2013 [16:15:48] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:15:39 UTC 2013 [16:15:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:48] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:15:41 UTC 2013 [16:15:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:15:47 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:15:50 UTC 2013 [16:15:58] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:15:55 UTC 2013 [16:16:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:08] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:16:02 UTC 2013 [16:16:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:16:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:18] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:16:12 UTC 2013 [16:16:28] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:16:18 UTC 2013 [16:16:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:16:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:16:28] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:16:21 UTC 2013 [16:16:28] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:16:23 UTC 2013 [16:16:28] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:16:23 UTC 2013 [16:16:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:38] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:16:28 UTC 2013 [16:16:38] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:16:28 UTC 2013 [16:16:38] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:16:28 UTC 2013 [16:16:38] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:16:30 UTC 2013 [16:16:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:38] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:16:33 UTC 2013 [16:16:39] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:16:37 UTC 2013 [16:16:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:16:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:16:48] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:16:43 UTC 2013 [16:16:48] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:16:47 UTC 2013 [16:16:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:16:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:17:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:08] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:17:04 UTC 2013 [16:17:08] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:17:06 UTC 2013 [16:17:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:18] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:19] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:20] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:17:20] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:17:10 UTC 2013 [16:17:20] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:17:12 UTC 2013 [16:17:21] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:17:17 UTC 2013 [16:17:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:38] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:17:30 UTC 2013 [16:17:38] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:17:30 UTC 2013 [16:17:38] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:17:30 UTC 2013 [16:17:48] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:17:40 UTC 2013 [16:17:48] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:17:43 UTC 2013 [16:17:58] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:17:50 UTC 2013 [16:17:58] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:17:54 UTC 2013 [16:17:58] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:17:55 UTC 2013 [16:18:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:08] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:18:02 UTC 2013 [16:18:08] PROBLEM - Host mw80 is DOWN: PING CRITICAL - Packet loss = 100% [16:18:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:18] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:18:16 UTC 2013 [16:18:21] paravoid: !!!! [16:18:34] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:18:18 UTC 2013 [16:18:34] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:18:34] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:18:34] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:18:19 UTC 2013 [16:18:34] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:18:21 UTC 2013 [16:18:34] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:18:25 UTC 2013 [16:18:34] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:18:26 UTC 2013 [16:18:34] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:34] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:18:27 UTC 2013 [16:18:38] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:18:30 UTC 2013 [16:18:38] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:18:30 UTC 2013 [16:18:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:38] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:18:35 UTC 2013 [16:18:38] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:18:36 UTC 2013 [16:18:48] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:18:39 UTC 2013 [16:18:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:18:48] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:18:45 UTC 2013 [16:18:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:18:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:19:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:08] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:18:59 UTC 2013 [16:19:08] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:19:01 UTC 2013 [16:19:08] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:19:05 UTC 2013 [16:19:08] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:19:06 UTC 2013 [16:19:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:20] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:19:20] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:21] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:19:10 UTC 2013 [16:19:21] RECOVERY - Host mw80 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [16:19:28] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:19:21 UTC 2013 [16:19:28] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:19:23 UTC 2013 [16:19:29] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:19:27 UTC 2013 [16:19:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:19:32 UTC 2013 [16:19:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:38] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:19:34 UTC 2013 [16:19:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:19:41 UTC 2013 [16:19:48] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:19:43 UTC 2013 [16:19:48] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:19:43 UTC 2013 [16:19:55] cmjohnson1: perhaps time to remove "puppet freshness" reporting as I don't think anyone cares about it nowadays [16:19:58] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:19:53 UTC 2013 [16:20:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:08] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:20:04 UTC 2013 [16:20:18] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:20:08 UTC 2013 [16:20:18] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:20:08 UTC 2013 [16:20:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:20:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:19] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:20:12 UTC 2013 [16:20:19] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:20:12 UTC 2013 [16:20:20] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:20:15 UTC 2013 [16:20:21] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:20:16 UTC 2013 [16:20:21] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:20:17 UTC 2013 [16:20:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:20:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:20:28] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:20:21 UTC 2013 [16:20:28] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:20:22 UTC 2013 [16:20:28] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:20:23 UTC 2013 [16:20:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:38] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:20:28 UTC 2013 [16:20:38] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:20:35 UTC 2013 [16:20:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:20:53] RobH: don't you agree? [16:20:57] azatoth: i agree [16:20:58] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:20:47 UTC 2013 [16:20:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:20:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:20:58] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:20:52 UTC 2013 [16:20:58] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:20:55 UTC 2013 [16:20:58] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:20:55 UTC 2013 [16:21:08] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:20:58 UTC 2013 [16:21:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:08] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:21:06 UTC 2013 [16:21:14] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:20:08 UTC 2013 [16:21:14] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:15] what. [16:21:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:18] cmjohnson1: is it easy to remove? [16:21:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:19] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:19] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:20] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:21:20] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:21] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:21] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:21:10 UTC 2013 [16:21:22] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:21:17 UTC 2013 [16:21:25] !log reedy synchronized wmf-config/InitialiseSettings.php [16:21:25] MatmaRex: it's fubar [16:21:28] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:21:18 UTC 2013 [16:21:28] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:21:18 UTC 2013 [16:21:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:21:38] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:21:28 UTC 2013 [16:21:38] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:21:28 UTC 2013 [16:21:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:38] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:21:31 UTC 2013 [16:21:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:48] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:21:39 UTC 2013 [16:21:58] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:21:51 UTC 2013 [16:21:58] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:21:52 UTC 2013 [16:21:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:21:53 UTC 2013 [16:21:58] PROBLEM - Host mw80 is DOWN: PING CRITICAL - Packet loss = 100% [16:21:58] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:21:55 UTC 2013 [16:21:58] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:21:56 UTC 2013 [16:22:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:08] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:22:00 UTC 2013 [16:22:08] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:22:00 UTC 2013 [16:22:08] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:22:01 UTC 2013 [16:22:09] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:09] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:22:03 UTC 2013 [16:22:10] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:22:04 UTC 2013 [16:22:10] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:22:07 UTC 2013 [16:22:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:22:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:18] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:18] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:19] azatoth: no idea but we'll figure it out [16:22:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:20] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:22:12 UTC 2013 [16:22:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:22:28] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:22:25 UTC 2013 [16:22:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:38] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:22:30 UTC 2013 [16:22:38] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:22:32 UTC 2013 [16:22:38] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:22:37 UTC 2013 [16:22:48] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:22:38 UTC 2013 [16:22:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:22:48] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:22:41 UTC 2013 [16:22:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:22:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:22:58] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:22:53 UTC 2013 [16:22:58] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:22:54 UTC 2013 [16:22:58] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:22:54 UTC 2013 [16:23:08] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:22:59 UTC 2013 [16:23:08] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:23:02 UTC 2013 [16:23:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:23:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:18] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:23:20] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:23:09 UTC 2013 [16:23:20] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:23:09 UTC 2013 [16:23:20] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:23:12 UTC 2013 [16:23:21] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:23:17 UTC 2013 [16:23:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:23:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:23:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:38] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:23:28 UTC 2013 [16:23:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:38] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:23:31 UTC 2013 [16:23:38] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:23:31 UTC 2013 [16:23:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:38] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:23:34 UTC 2013 [16:23:38] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:23:35 UTC 2013 [16:23:48] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:23:38 UTC 2013 [16:23:48] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:23:38 UTC 2013 [16:23:48] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:23:38 UTC 2013 [16:23:48] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:23:40 UTC 2013 [16:23:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:23:48] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:23:42 UTC 2013 [16:23:48] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:23:43 UTC 2013 [16:23:57] ^demon: it's currenlt puppet freshness frenzy [16:24:04] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:23:49 UTC 2013 [16:24:04] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:24:04] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:24:04] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:23:52 UTC 2013 [16:24:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:08] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:24:06 UTC 2013 [16:24:08] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:24:06 UTC 2013 [16:24:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:18] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:19] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:20] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:24:20] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:24:09 UTC 2013 [16:24:20] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:24:11 UTC 2013 [16:24:21] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:24:15 UTC 2013 [16:24:28] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:24:20 UTC 2013 [16:24:28] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:24:21 UTC 2013 [16:24:28] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:24:24 UTC 2013 [16:24:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:38] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:24:31 UTC 2013 [16:24:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:38] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:24:33 UTC 2013 [16:24:48] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:24:39 UTC 2013 [16:24:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:24:40 UTC 2013 [16:24:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:48] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:24:41 UTC 2013 [16:24:57] meh, better to fix it than remove it. [16:24:58] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:24:50 UTC 2013 [16:24:58] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:24:57 UTC 2013 [16:25:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:08] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:25:02 UTC 2013 [16:25:08] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:25:03 UTC 2013 [16:25:08] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:25:04 UTC 2013 [16:25:08] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:25:05 UTC 2013 [16:25:09] New patchset: Reedy; "Move AFTv5 feedback disabling for dewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71635 [16:25:09] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:25:06 UTC 2013 [16:25:09] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:25:07 UTC 2013 [16:25:18] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:25:08 UTC 2013 [16:25:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:25:19] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:19] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:20] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:21] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:22] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:25:10 UTC 2013 [16:25:22] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:25:13 UTC 2013 [16:25:22] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:25:14 UTC 2013 [16:25:28] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:25:18 UTC 2013 [16:25:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:25:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:25:28] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:25:22 UTC 2013 [16:25:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:33] RobH: how? [16:25:38] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:25:35 UTC 2013 [16:25:48] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:25:38 UTC 2013 [16:25:48] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:25:40 UTC 2013 [16:25:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:25:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:50] RobH: I don't even know where the config is located at [16:25:58] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:25:49 UTC 2013 [16:25:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:25:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:25:58] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:25:51 UTC 2013 [16:25:58] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:25:52 UTC 2013 [16:25:58] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:25:53 UTC 2013 [16:25:58] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:25:57 UTC 2013 [16:26:02] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71635 [16:26:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:08] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:25:59 UTC 2013 [16:26:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:08] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:26:06 UTC 2013 [16:26:08] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:26:06 UTC 2013 [16:26:08] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:26:06 UTC 2013 [16:26:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:26:18] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:18] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:19] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:26:13 UTC 2013 [16:26:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:26:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:26:28] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:26:22 UTC 2013 [16:26:29] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:26:26 UTC 2013 [16:26:29] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:26:26 UTC 2013 [16:26:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:38] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:26:28 UTC 2013 [16:26:38] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:26:29 UTC 2013 [16:26:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:38] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:26:30 UTC 2013 [16:26:38] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:26:30 UTC 2013 [16:26:38] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:26:31 UTC 2013 [16:26:39] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:39] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:26:34 UTC 2013 [16:26:40] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:26:35 UTC 2013 [16:26:40] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:26:36 UTC 2013 [16:26:44] RobH: is the configuration in some git somewhere? [16:26:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:26:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:26:48] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:26:42 UTC 2013 [16:26:48] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:26:44 UTC 2013 [16:26:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:26:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:26:58] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:26:51 UTC 2013 [16:26:58] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:26:53 UTC 2013 [16:26:58] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:26:57 UTC 2013 [16:27:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:08] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:26:59 UTC 2013 [16:27:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:08] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:27:03 UTC 2013 [16:27:18] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:27:08 UTC 2013 [16:27:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:19] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:19] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:20] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:20] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:21] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:22] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:22] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:27:09 UTC 2013 [16:27:22] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:27:09 UTC 2013 [16:27:24] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:27:12 UTC 2013 [16:27:24] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:27:12 UTC 2013 [16:27:24] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:27:17 UTC 2013 [16:27:29] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:27:18 UTC 2013 [16:27:29] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:27:29] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:27:24 UTC 2013 [16:27:29] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:27:25 UTC 2013 [16:27:29] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:27:25 UTC 2013 [16:27:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:38] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:27:32 UTC 2013 [16:27:41] Azatath I can disable freshness check for now [16:27:43] urgh, noisy. [16:27:48] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:27:41 UTC 2013 [16:27:48] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:27:43 UTC 2013 [16:27:48] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:27:44 UTC 2013 [16:27:48] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:27:45 UTC 2013 [16:27:48] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:27:46 UTC 2013 [16:27:54] robh: going to disable freshness check [16:27:58] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:27:48 UTC 2013 [16:27:58] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:27:48 UTC 2013 [16:27:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:27:58] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:27:50 UTC 2013 [16:27:58] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:27:52 UTC 2013 [16:27:58] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:27:53 UTC 2013 [16:27:59] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:27:53 UTC 2013 [16:27:59] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:27:56 UTC 2013 [16:28:07] if you do you should admin log it and possibly send email to ops list [16:28:08] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:27:58 UTC 2013 [16:28:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:11] cuz its an active check [16:28:18] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:28:07 UTC 2013 [16:28:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:28:19] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:19] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:20] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:20] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:20] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:21] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:28:09 UTC 2013 [16:28:21] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:28:10 UTC 2013 [16:28:22] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:28:13 UTC 2013 [16:28:23] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:28:15 UTC 2013 [16:28:29] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:28:30] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:28:23 UTC 2013 [16:28:30] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:28:23 UTC 2013 [16:28:30] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:28:24 UTC 2013 [16:28:30] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:28:26 UTC 2013 [16:28:33] robh: okay [16:28:36] RobH: I wonder if anyone has cared about the freshness check since 2003 [16:28:38] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:28:29 UTC 2013 [16:28:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:38] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:28:31 UTC 2013 [16:28:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:38] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:28:37 UTC 2013 [16:28:39] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:28:37 UTC 2013 [16:28:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:28:48] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:28:42 UTC 2013 [16:28:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:28:44 UTC 2013 [16:28:51] AzaToth: we do [16:28:58] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:28:48 UTC 2013 [16:28:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:28:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:28:51 UTC 2013 [16:28:58] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:28:52 UTC 2013 [16:28:58] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:28:53 UTC 2013 [16:28:58] !log disabling puppet freshness check [16:28:58] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:28:53 UTC 2013 [16:28:59] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:28:55 UTC 2013 [16:28:59] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:28:55 UTC 2013 [16:28:59] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:28:56 UTC 2013 [16:29:00] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:28:57 UTC 2013 [16:29:05] we care about puppet freshness, many of us check servermon for it [16:29:08] Logged the message, Master [16:29:08] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:28:58 UTC 2013 [16:29:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:08] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:29:01 UTC 2013 [16:29:09] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:09] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:29:04 UTC 2013 [16:29:10] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:29:05 UTC 2013 [16:29:15] New patchset: Tim Landscheidt; "Add mariadb-client to toollabs's exec_environ" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71636 [16:29:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:19] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:19] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:29:19] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:20] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:20] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:29:21] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:22] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:22] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:29:15 UTC 2013 [16:29:23] the issue is the flaps, there are longstanding fialed checks that are useful. [16:29:34] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:29:18 UTC 2013 [16:29:34] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:29:18 UTC 2013 [16:29:34] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:29:34] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:29:34] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:29:20 UTC 2013 [16:29:34] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:29:20 UTC 2013 [16:29:34] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:34] and something is up cuz it didnt used to be this bad. [16:29:38] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:29:29 UTC 2013 [16:29:38] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:29:29 UTC 2013 [16:29:38] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:29:29 UTC 2013 [16:29:38] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:29:30 UTC 2013 [16:29:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:38] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:29:34 UTC 2013 [16:29:39] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:29:36 UTC 2013 [16:29:48] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:29:39 UTC 2013 [16:29:48] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:29:39 UTC 2013 [16:29:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:29:39 UTC 2013 [16:29:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:29:48] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:29:47 UTC 2013 [16:29:52] New patchset: Cmjohnson; "disabling puppet freshness -causing lots of noise on IRC" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71637 [16:29:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:29:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:29:58] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:29:53 UTC 2013 [16:29:58] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:29:54 UTC 2013 [16:29:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:29:56 UTC 2013 [16:29:58] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:29:56 UTC 2013 [16:29:59] robh: yeah something is up [16:30:00] RobH: I've seen this for some weeks [16:30:08] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:29:57 UTC 2013 [16:30:08] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:29:57 UTC 2013 [16:30:08] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:29:58 UTC 2013 [16:30:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:08] probably snmptt [16:30:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:09] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:29:59 UTC 2013 [16:30:09] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:10] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:30:01 UTC 2013 [16:30:10] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:11] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:30:02 UTC 2013 [16:30:11] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:30:02 UTC 2013 [16:30:12] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:30:06 UTC 2013 [16:30:15] everyone keeps papering over the issue rather than fixing it [16:30:18] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:30:08 UTC 2013 [16:30:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:19] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:19] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:20] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:30:20] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:21] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:21] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:22] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:22] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:30:23] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:23] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:30:15 UTC 2013 [16:30:23] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:30:16 UTC 2013 [16:30:29] last time they said it was because paravoid did something [16:30:29] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:30:18 UTC 2013 [16:30:29] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:30:30] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:30:30] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:30:20 UTC 2013 [16:30:30] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:30:21 UTC 2013 [16:30:30] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:30] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:30:27 UTC 2013 [16:30:38] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:30:27 UTC 2013 [16:30:38] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:30:29 UTC 2013 [16:30:38] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:30:29 UTC 2013 [16:30:39] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:39] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:39] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:30:32 UTC 2013 [16:30:39] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:30:37 UTC 2013 [16:30:39] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:30:37 UTC 2013 [16:30:48] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:30:39 UTC 2013 [16:30:48] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:30:39 UTC 2013 [16:30:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:30:48] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:30:43 UTC 2013 [16:30:57] nope, it's not me [16:30:58] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:30:48 UTC 2013 [16:30:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:30:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:30:58] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:30:50 UTC 2013 [16:30:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:30:50 UTC 2013 [16:30:58] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:30:51 UTC 2013 [16:30:59] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:30:52 UTC 2013 [16:30:59] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:30:53 UTC 2013 [16:31:00] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:30:54 UTC 2013 [16:31:00] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:30:54 UTC 2013 [16:31:01] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:30:56 UTC 2013 [16:31:01] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:30:57 UTC 2013 [16:31:08] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:30:58 UTC 2013 [16:31:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:08] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:30:59 UTC 2013 [16:31:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:09] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:31:01 UTC 2013 [16:31:09] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:10] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:31:06 UTC 2013 [16:31:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:18] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:18] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:19] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:19] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:20] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:31:20] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:21] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:21] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:22] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:31:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:23] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:31:09 UTC 2013 [16:31:24] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:31:09 UTC 2013 [16:31:24] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:31:11 UTC 2013 [16:31:24] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:31:12 UTC 2013 [16:31:26] paravoid: can only make a educated guess it stores all info in a queue and suddenly releases [16:31:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:31:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:31:29] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:31:19 UTC 2013 [16:31:29] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:31:19 UTC 2013 [16:31:29] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:31:20 UTC 2013 [16:31:29] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:31:20 UTC 2013 [16:31:29] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:31:24 UTC 2013 [16:31:30] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:30] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:31:27 UTC 2013 [16:31:30] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:31:27 UTC 2013 [16:31:36] s/educated/uneducated/ [16:31:38] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:31:29 UTC 2013 [16:31:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:38] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:31:33 UTC 2013 [16:31:48] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:31:37 UTC 2013 [16:31:48] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:31:39 UTC 2013 [16:31:48] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:31:40 UTC 2013 [16:31:48] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:31:40 UTC 2013 [16:31:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:31:49] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:31:41 UTC 2013 [16:31:49] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:31:42 UTC 2013 [16:31:50] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:31:43 UTC 2013 [16:31:50] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:31:43 UTC 2013 [16:31:51] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:31:44 UTC 2013 [16:31:54] robh: can you merge that change..my gerrit login is not working [16:31:58] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:31:48 UTC 2013 [16:31:58] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:31:49 UTC 2013 [16:31:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:31:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:31:58] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:31:55 UTC 2013 [16:32:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:08] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 16:32:03 UTC 2013 [16:32:09] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:32:06 UTC 2013 [16:32:09] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 16:32:06 UTC 2013 [16:32:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:19] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:20] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:20] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:21] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:21] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:21] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:22] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:32:22] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 16:32:10 UTC 2013 [16:32:23] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Tue Jul 2 16:32:11 UTC 2013 [16:32:24] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 16:32:13 UTC 2013 [16:32:24] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Tue Jul 2 16:32:16 UTC 2013 [16:32:28] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 16:32:18 UTC 2013 [16:32:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:32:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:28] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Tue Jul 2 16:32:19 UTC 2013 [16:32:28] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 16:32:19 UTC 2013 [16:32:28] RECOVERY - Puppet freshness on cp1031 is OK: puppet ran at Tue Jul 2 16:32:20 UTC 2013 [16:32:29] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Tue Jul 2 16:32:20 UTC 2013 [16:32:30] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Tue Jul 2 16:32:20 UTC 2013 [16:32:30] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Tue Jul 2 16:32:21 UTC 2013 [16:32:30] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:32:22 UTC 2013 [16:32:31] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:32:25 UTC 2013 [16:32:31] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:38] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:32:35 UTC 2013 [16:32:39] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:32:36 UTC 2013 [16:32:39] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:32:37 UTC 2013 [16:32:48] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:32:38 UTC 2013 [16:32:48] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 16:32:39 UTC 2013 [16:32:48] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:32:39 UTC 2013 [16:32:48] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:32:39 UTC 2013 [16:32:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:49] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Tue Jul 2 16:32:43 UTC 2013 [16:32:49] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 16:32:43 UTC 2013 [16:32:50] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 16:32:44 UTC 2013 [16:32:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:58] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 16:32:51 UTC 2013 [16:32:58] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 16:32:54 UTC 2013 [16:32:58] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 16:32:55 UTC 2013 [16:32:58] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 16:32:57 UTC 2013 [16:33:08] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:08] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:08] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:08] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:08] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Tue Jul 2 16:33:03 UTC 2013 [16:33:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:18] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:18] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [16:33:19] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:19] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:20] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:20] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:21] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:22] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:22] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:22] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:33:28] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [16:33:28] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [16:33:29] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:38] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:48] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:48] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [16:33:58] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:35:06] cmjohnson1: /var/lib/icinga/spool/checkresults is that dir filled? [16:35:08] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Tue Jul 2 16:35:05 UTC 2013 [16:35:32] New review: Se4598; "This change partly doesn't do what it should: It does not disable submission on articles with the ca..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71546 [16:35:56] seems it might be over for today though [16:35:58] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [16:36:18] New review: RobH; "Chris says this will work, so lets try it." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71637 [16:36:18] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71637 [16:36:38] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Tue Jul 2 16:36:37 UTC 2013 [16:37:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:37:35] cmjohnson1: ok, its merged on sockpuppet and neon is mid puppet run now [16:38:47] okay..thx [16:39:58] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Tue Jul 2 16:39:55 UTC 2013 [16:40:38] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:40:48] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Tue Jul 2 16:40:46 UTC 2013 [16:41:08] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Tue Jul 2 16:41:02 UTC 2013 [16:41:18] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:41:18] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:41:58] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Tue Jul 2 16:41:47 UTC 2013 [16:42:18] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:18] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Tue Jul 2 16:43:09 UTC 2013 [16:59:38] cmjohnson1: hey [16:59:45] hey [16:59:53] cmjohnson1: grrr, the biggest problem is that /var/spool/snmptt is getting filled and not read from [16:59:58] uncertain why that is :( [17:00:02] also, are yo at the dc ? [17:00:02] yeah..i was just reading that ticket [17:00:11] with the replacement of row switch 8, we need to move the uplinks ? [17:00:22] in eqiad? [17:00:27] yep [17:00:32] no..i am in tampa this week [17:00:35] oh yeah [17:00:36] i forgot [17:01:07] wanna create a ticket so we can do this next week [17:01:11] well, next week all the sfp+'s should be there anyways … massive row switch uplink party ? [17:01:17] hehe, i was just in rt :) [17:01:21] woo hoo! [17:01:57] rt 5405 [17:03:44] cool...i linked my ticket with 5311 (neon inode issue) [17:06:15] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:15] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [17:06:25] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [17:06:35] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [17:06:35] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [17:06:35] RECOVERY - Host mw80 is UP: PING WARNING - Packet loss = 93%, RTA = 26.57 ms [17:06:45] PROBLEM - twemproxy process on mw80 is CRITICAL: Timeout while attempting connection [17:06:45] PROBLEM - SSH on mw80 is CRITICAL: Connection timed out [17:06:45] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:45] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:45] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [17:06:45] PROBLEM - RAID on mw80 is CRITICAL: Timeout while attempting connection [17:06:55] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:55] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:55] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:55] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:55] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:55] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:56] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:56] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:57] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:57] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:57] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:03] it's sadly all the same issue [17:07:05] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [17:07:05] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:05] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [17:07:52] grrr [17:08:25] obviously disabling puppet freshness check didn't work either [17:08:34] or i missed something [17:09:05] PROBLEM - Host mw80 is DOWN: PING CRITICAL - Packet loss = 100% [17:14:50] New patchset: CSteipp; "Move loginwiki to wmf9" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71639 [17:15:58] Reedy you around? [17:16:25] PROBLEM - Varnish HTCP daemon on cp1050 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (vhtcpd), args vhtcpd [17:19:00] New patchset: Ottomata; "Adding spetrea to admins::restricted so he has an account on bastion hosts." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/69313 [17:24:41] csteipp: Aye [17:26:02] Reedy, just wondering if I did https://gerrit.wikimedia.org/r/71639 right... and then I just run sync-wikiversion? [17:26:29] (bumping login wiki to wmf9.. we were talking about it on the call yesterday) [17:26:41] Yup, exactly [17:26:49] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71639 [17:26:58] Thanks! [17:27:25] PROBLEM - DPKG on analytics1006 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [17:28:34] !log csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: bumping loginwiki to wmf9 [17:28:43] Logged the message, Master [17:29:22] grr, crashed irc client -- did i miss anything important re:snmp ideas ? [17:29:43] greg-g: done ^ [17:29:51] Reedy: you probably missed it since you couldn't get in. We're going to move login. to group0/phase0/whatever you wanna call it so it is always the latest deployed version [17:29:59] csteipp: sweet [17:30:34] New patchset: Ottomata; "Adding support for arbitrary ganglia output writer settings" [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/71640 [17:31:11] * greg-g updates Deployments/MW Roadmap wiki pages with that change [17:32:35] ahha! strace!!! [17:32:40] "sh -c /usr/lib/nagios/plugins/eventhandlers/submit_check_result db66.pmtpa.wmnet "Puppet freshness" 0 "puppet ran at `date`"" [17:32:48] it's waiting on that process which froze at the time the files started to back up [17:32:52] now, why did that freeze ? [17:32:55] New patchset: Ottomata; "Adding support for arbitrary ganglia output writer settings" [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/71640 [17:33:42] Change merged: Ottomata; [operations/puppet/jmxtrans] (master) - https://gerrit.wikimedia.org/r/71640 [17:36:17] New patchset: Ottomata; "Updating jmxtrans git submodule to support arbitrary ganglia output writer settings" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71642 [17:37:11] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71642 [17:37:17] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 17:37:08 UTC 2013 [17:37:17] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 17:37:09 UTC 2013 [17:37:17] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 17:37:11 UTC 2013 [17:37:17] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 17:37:11 UTC 2013 [17:37:17] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 17:37:12 UTC 2013 [17:37:17] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 17:37:15 UTC 2013 [17:37:27] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [17:37:28] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [17:37:37] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [17:37:47] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 17:37:46 UTC 2013 [17:37:47] !log replacing disk on db44 [17:37:57] Logged the message, Master [17:37:57] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 17:37:53 UTC 2013 [17:37:57] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 17:37:53 UTC 2013 [17:38:07] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [17:38:07] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [17:38:07] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [17:38:11] awtf [17:38:12] ok [17:38:15] this is so fucking weird [17:38:27] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [17:38:28] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [17:38:29] so i strace the lowest child process of the little process tree that is stuck [17:38:37] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [17:38:45] it then exits, and the whole tree is completed [17:39:27] RECOVERY - DPKG on analytics1006 is OK: All packages OK [17:44:03] New review: Lcarr; "i would like to try and fix the core result - looks like submit_check_result may be hanging for some..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/71149 [17:44:07] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 17:44:06 UTC 2013 [17:44:28] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [17:44:47] New patchset: Lcarr; "fixed neon Revert "disabling puppet freshness -causing lots of noise on IRC"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71644 [17:45:11] heh, no RT duty [17:47:03] greg-g .. Ryan_Lane is on duty [17:47:11] I am indeed [17:47:26] * greg-g jokes [17:47:30] !log replacing disks slot5 & slot6 on db45 [17:47:39] Logged the message, Master [17:47:57] greg-g: was there something you needed? [17:48:09] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71644 [17:48:38] Ryan_Lane: not at all, just noticed the momemtary emptiness [17:48:47] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Tue Jul 2 17:48:44 UTC 2013 [17:48:57] emptiness? :) [17:49:08] well, "?" [17:49:37] PROBLEM - Puppet freshness on cp1025 is CRITICAL: No successful Puppet run in the last 10 hours [17:49:47] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Tue Jul 2 17:49:41 UTC 2013 [17:50:05] still don't understand [17:50:07] PROBLEM - Puppet freshness on cp1023 is CRITICAL: No successful Puppet run in the last 10 hours [17:50:07] * Ryan_Lane shrugs [17:50:37] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 17:50:32 UTC 2013 [17:51:28] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [17:52:06] Ryan_Lane, https://rt.wikimedia.org/Ticket/Display.html?id=5400 could do with checking if there are any mchenry forwarders for that email address - apparently there was a forwarder when Keegan left so that could be what's causing the issue [17:52:17] RECOVERY - Puppet freshness on cp1035 is OK: puppet ran at Tue Jul 2 17:52:14 UTC 2013 [17:52:17] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Tue Jul 2 17:52:14 UTC 2013 [17:52:27] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 10 hours [17:52:58] ahha [17:53:07] those are all because they're still live but in decommissioning.pp [17:53:07] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Tue Jul 2 17:53:00 UTC 2013 [17:53:07] PROBLEM - Puppet freshness on cp1035 is CRITICAL: No successful Puppet run in the last 10 hours [17:53:27] PROBLEM - Puppet freshness on cp1021 is CRITICAL: No successful Puppet run in the last 10 hours [17:54:20] New patchset: Lcarr; "no need for echocmd when bin/bash has echo built in" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71645 [17:54:57] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Jul 2 17:54:48 UTC 2013 [17:55:07] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [17:57:02] New patchset: BBlack; "add dequeue stat" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71647 [17:57:03] New patchset: BBlack; "bugfix: parser incompletion was causing queue backlog..." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71648 [17:57:03] New patchset: BBlack; "NEWS update / version bump for 0.0.8" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71649 [17:57:30] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71647 [17:58:33] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71648 [17:58:53] Change merged: BBlack; [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/71649 [17:59:49] whee, we've got an on-duty person! Ryan_Lane, can you please copy solr1001:/var/log/jetty/*.stderrout.log* to my home dir on fenari? [18:03:48] New patchset: BBlack; "Merge branch 'master' into debian" [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71651 [18:03:48] New patchset: BBlack; "bump pkg version" [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71652 [18:04:00] Change merged: BBlack; [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71651 [18:04:10] Change merged: BBlack; [operations/software/varnish/vhtcpd] (debian) - https://gerrit.wikimedia.org/r/71652 [18:06:02] Is there a way to clear i18n caches on testwiki other than running scap? [18:06:17] MaxSem: your home directory where? [18:06:23] this is 4G of logs [18:06:34] oh shi... [18:06:44] is it for the last 7 days? [18:06:49] yes [18:06:55] !log upgraded vhtcpd to 0.0.8 on brewster [18:06:56] how's the puppet freshness thingi going? [18:07:02] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:02] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:02] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [18:07:02] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [18:07:06] Logged the message, Master [18:07:09] * AzaToth rests case [18:07:12] PROBLEM - Puppet freshness on cerium is CRITICAL: No successful Puppet run in the last 10 hours [18:07:12] Ryan_Lane, then just one oldest file [18:07:12] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:22] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:22] RECOVERY - Varnish HTCP daemon on cp1050 is OK: PROCS OK: 1 process with UID = 111 (vhtcpd), args vhtcpd [18:07:32] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [18:07:32] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:35] MaxSem: to which homedir? fenari? [18:07:42] PROBLEM - Puppet freshness on cp1026 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:42] PROBLEM - Puppet freshness on cp1028 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:42] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:42] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:42] PROBLEM - Puppet freshness on cp1032 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:42] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:43] PROBLEM - Puppet freshness on cp1033 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:43] PROBLEM - Puppet freshness on cp1034 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:43] PROBLEM - Puppet freshness on cp1030 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:44] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [18:07:48] fenari [18:07:52] PROBLEM - Puppet freshness on cp1022 is CRITICAL: No successful Puppet run in the last 10 hours [18:07:52] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [18:07:52] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [18:08:49] MaxSem: done [18:08:59] thanks a lot!:) [18:09:02] yw [18:12:12] RECOVERY - RAID on db44 is OK: OK: State is Optimal, checked 2 logical device(s) [18:19:21] Thehelpfulone: seems it's an issue in OIT's LDAP [18:19:52] Ryan_Lane: heya, so, uh, like jon r, I too have forgotten my RT password (I guess I didn't have lastpass remember it, then the upgrade). Mind reseting it for me? greg@wikimedia [18:20:04] if I can figure out how [18:20:09] :) [18:20:48] you can't have it send you one? [18:21:46] I don't see any options in the interface for resetting passwords [18:22:26] greg-g, you should be able to password reset through email? [18:23:41] Ryan_Lane: yeah, it be weird like that [18:23:50] Thehelpfulone: how? what encantation do I use? [18:24:09] RT: pervasive but horrible. Kinda like ADP [18:24:24] hmm there was an option before O_O [18:24:39] one sec [18:24:50] there was... [18:25:24] we just upgarded recently. Apparently the ability to reset your password was a bug [18:27:37] Ryan_Lane, if you login through root you should be able to do it somehow [18:27:51] but I'm trying to figure out how the reset by email disappeared [18:28:38] yeah. found it [18:29:01] where's jon? [18:29:03] can you enable password reset through the root too? [18:31:56] Ryan_Lane, it looks like it doesn't exist in RT 4? http://requesttracker.wikia.com/wiki/PasswordReset is a separate thing to install [18:32:47] hilarious [18:32:58] seriously, what is the reasoning? [18:43:30] oh hey, qchris, since we are both here [18:43:35] should I merge this? [18:43:36] https://gerrit.wikimedia.org/r/#/c/71248/ [18:43:41] Hi. [18:43:41] oh [18:43:42] sorry [18:43:49] reading your comment, didnt' realized you had commented since yesterday [18:43:50] sorry [18:44:09] But for me it's ok to merge it. [18:44:15] <^d> Fine by me too [18:44:23] does gerrit replication only replicate master? [18:44:30] <^d> https://gerrit.wikimedia.org/r/#/c/71244/ would be nice too, if we're at it [18:44:45] I'd really like to get this merged while ^d is here, so if replication fails, we have someone to look at the logs. [18:45:17] ottomata: No. By default all refs. [18:45:39] ottomata: But for the cdh4 repo, we'd only be taking heads. [18:45:44] And tags. [18:46:00] qchris: ottomata should have access to logs too? [18:46:31] jeremyb: Ok. The more eyes the better :-) [18:46:34] Ryan_Lane, https://rt.wikimedia.org/Ticket/Display.html?id=5407 ticket for adding password reset back - that may need discussion, maybe it was added for extra security or something? [18:47:11] well missing password reset means no new self serve peoples [18:48:47] RECOVERY - RAID on db45 is OK: OK: State is Optimal, checked 2 logical device(s) [18:49:22] qchris: , i should merge 71244? [18:49:36] That'd be great. Yes, please. [18:50:00] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71244 [18:50:15] ottomata: Yippie! Thanks. [18:50:27] k great done [18:50:41] qchris, for puppet-cdh4 [18:50:50] what if I first manually merge from gerrit to github and push to github? [18:50:55] get everythign in sync exactly [18:51:10] That would do the trick. [18:51:28] ok, lemme see if I can do that [18:51:28] But I wasn't sure which way to merge ... So I wanted to leave that to you. [18:51:33] k [18:51:43] ottomata: the v0.1 would probably need the same treatment. [18:51:45] bblack, ping [18:52:06] v0.1 branch should be the same on both, I think [18:52:10] will check [18:52:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [18:54:22] yurik: pong [18:54:51] bblack, hi! I just replied to your email, but have you had a chance to see re SSL issue in the rt ticket? [18:54:53] yurik: the two separate IP->string DBs isn't a problem, as two different JSON files loaded via vcl_init in two different VCLs [18:55:26] oh, btw, i couldn't find you in https://office.wikimedia.org/wiki/Contact_list -- pls add :) [18:55:58] yurik: and in general a 3rd database for opera also isn't an issue, although I don't really get the push/pop bit (which can probably stay in VCL anyways) [18:56:41] Wait, am I still on RT duty?! [18:56:59] crapcrapcrap, I haven't even looked at RT [18:57:28] Coren, don't worry, the site has been down for a few hours, its all good ;) [18:57:53] yurik: although with 3 databases and related conditionals, it might be a pain to use a separate VCL for each. I could revisit it such that the databases are indexed instead and can all be used in the same VCL file [18:58:12] * Coren thought someone else took over on monday. [18:58:38] Oh, yes. Someone /has/ [18:58:59] :-P [19:00:14] bblack, i am not exactly sure how its connected to VCL files - not sure how the plugin mechanism works in varnish. Does it mean that each DB has to have a separate VCL file? Or can it be used as a lib function call from anywhere? [19:01:35] yurik: the way the netmapper vmod works now, it uses "per-VCL" data for the database stuff. So: vcl_init { netmapper.init("opera.json"); } recv_recv { whatever = netmapper.map("127.0.0.1") } [19:01:42] Coren, that graph looks like someone decided to show an inappropriate gesture to the world ;) [19:01:52] hmm, qchris [19:02:00] New patchset: Tim Landscheidt; "Add mariadb-client to toollabs's exec_environ" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71636 [19:02:01] New patchset: Tim Landscheidt; "Add packages for Checkwiki to toollabs's exec_environ" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71657 [19:02:03] hmm, ottomata :-) [19:02:03] yurik: and apparently you can have multiple separate VCLs with their own intialization and database that way [19:02:11] some of the commits from v0.1 are merged pull requests [19:02:20] which gerrit doesn't like, because I am not the committer [19:02:39] remote: ERROR: committer email address tom@supertom.com [19:02:40] remote: ERROR: does not match your user account. [19:02:43] yurik: but we can probably do something else and give them indices of some kind [19:02:58] ottomata: Lets see who can forge them. [19:03:03] v0.1 is kind of deprecated, but I want to keep it in place for current users and for posterity [19:03:09] it is more featureful than master (v0.2) right now [19:03:17] bblack, so its not like you can create an instance of the netmapper. Sure, indicies sound like a good solution [19:03:48] yurik: right, the current code relies on the idea that Varnish will do the instancing, per-VCL-file. [19:04:44] bblack, but my concern is how to better deal with string parsing in varnish - because the IP that gets passed to the netmapper might come from the .ip structure or from the XFF header [19:04:46] we could just do some integer stuff, e.g. netmapper.load(0, "opera.json"); netmapper.load(1, "foo.json"); netmapper.map(0, "127.0.0.1") [19:05:03] bleh, but faster :) [19:05:26] i guess with only a few indicies, it makes sense, no need for a full blown dictionary [19:05:56] ottomata: It seems like no one can forge committer addresses in gerrit's cdh4 repo. Looks like you should get them to get this done. Let's ask ^d. [19:05:59] I can make it a dictionary with labels if you want, I just tend to err on the side of not making the machine do things we can do easily :) [19:06:19] qchris, should I try to give myself permissions to force push those commits, just so the branches match…or should I try to rewrite the committer on the history? [19:06:23] probably not option 2 :p [19:06:24] qchris: afaik that's a permission [19:06:27] yuril: "" + client.ip converts to string form, and the vmod requires string form, so that part's easy [19:06:32] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [19:06:32] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [19:06:33] <^d> qchris, ottomata: Everyone should have forge author. Forge committer I don't grant usually. [19:06:42] ^d: ottomata wants to bring github pullrequests to a gerrit repo (cdh4), can we grant him forgeCommiter for this? [19:06:42] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [19:06:42] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [19:06:42] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [19:06:52] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [19:07:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [19:07:09] bblack, although come to think of it, if the label is the name of the file, and each time it will re-use the string pointer, it might be fairly fast [19:07:19] and could load dict on the fly [19:07:27] qchris: there are permissions "Forge commiter identity" and "Forge author identity" [19:07:29] without having a separate init section [19:07:36] yurik: I'm not sure it's sane to use string pointers as indices across the vcl<->vmod barrier [19:07:40] AzaToth: Yes. Both exist. [19:07:55] New review: coren; "LGM (eew. command-line mysql from tools?)" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71636 [19:07:55] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71636 [19:07:57] and forge server identity which I've no idea what it implies [19:08:02] in this case I think I'd need committer [19:08:13] since the committer email is what gerrit is complaining about [19:08:13] bblack, no no, i meant that any string comparison will always start as if pointers equal, or if content equals, then string is the same [19:08:29] New review: coren; "More prelness." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71657 [19:08:30] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71657 [19:08:37] ^d: So how can ottomata push the commits of others if he cannot forgeCommitter? Should he insert himself as commiter and take credit for it? [19:08:47] ^d, which maybe is ok for this just temporarily? I wouldn't do this regularly [19:08:56] <^d> Temporarily? Totally. [19:08:57] well, committer doesn't really mean credit [19:08:59] yurik: at that point it may as well be a hash. either that or we're doing strcmp() x N databases [19:09:01] author would still be correct [19:09:04] <^d> I do that all the time when I import existing history. [19:09:12] ^d ok. [19:09:17] but, it would be different than what the github history says if I edited the committer manually [19:09:29] otherwise I'd just edit the history, since in this case I kinda am the committer (to gerrit) [19:09:39] but easier for now to do tempory forge commit [19:09:41] to get histories to match [19:09:43] before replication [19:09:45] ok, will try that [19:10:13] bblack, sure, whichever is easier :) besides, index #0 will be used all the time, whereas the other ones might be used less frequently [19:10:27] yurik: the rest (the push/pop/ssl stuff, IP object-vs-string) I think can be trivially solved at the VCL layer and we can get to that. let me go ahead and update the vmod to not need multiple VCLs to do it first, though. [19:10:56] great, that worked [19:12:13] bblack, sounds good. I just wasn't sure if VCL will make it easy to do things like "use the first value before comma of the XFF header in netmapper, and if some condition, take the second value from XFF and use that for netmapper [19:12:26] yurik: tell you what, as a compromise, I'll give it string database stuff without explicit load, and just iterate + strcmp, which is fine for like 3 databases. if someone later uses it for 400 databases, they can add the hash table :P [19:12:33] qchris: let's do it! i'm going to merge this, ok? [19:12:41] oooooo, wait no [19:12:43] i have to do master too [19:12:45] hold on! [19:12:50] ottomata: The master branches do not yet agrree, do they? [19:12:52] bblack, nah, lets keep an integer index :) [19:12:57] ottomata: Yes :-) [19:13:16] bblack, unless you gain some benefit from on-the-fly loading if the filename is used [19:13:40] wooooooo so many conflicts! [19:14:08] bblack, but i don't think its a good idea to do it on the fly - there are few databases, and we don't want to stall on realtime request [19:14:18] yeah ok [19:14:20] better preload everything [19:14:21] integers it is! [19:14:28] cool [19:14:53] bblack, but do you think we can easily do string parsing in varnish as i wrote above? [19:15:17] splitting XFF header into subparts [19:15:22] yurik: probably [19:15:40] the important thing is it's relatively static logic, I'm sure there's a way [19:15:50] ok [19:15:52] thx! [19:18:49] fyi, I am going to run scap [19:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [19:23:28] ok qchris, more questions [19:23:34] not sure if I am doing this properly [19:23:37] ottomata: shoot. [19:23:44] so gerrit/master is ahead of origin/master [19:23:45] or [19:23:47] github/master [19:23:47] really [19:23:48] so [19:23:51] i do [19:24:00] git merge -Xtheirs gerrit/master [19:24:05] (into my github master branch) [19:24:17] i get some conflicts, but -Xtheirs actually does pretty good [19:24:20] so they are easy to resolve [19:24:23] so I commit [19:24:27] but that creates an actual merge commit [19:24:43] so now even though the content of gerrit/master and github/master should be the same [19:24:47] they aren't at the same commit [19:24:54] because the merge commit is an extra commit [19:25:01] I cannot —ff-only [19:25:18] But you're having the merge commit after the old heads of the gerrit and github master? [19:26:00] So the old github master can ff to your new head, and also the old gerrit master can ff to your new head? [19:26:33] ottomata: can you add a 'verified' comment to https://gerrit.wikimedia.org/r/#/c/70928/ so that it will merge? [19:26:43] HMMMM [19:26:55] mark already gave it +2 [19:27:09] drdee, you want me to merge it? [19:27:13] or just +2 verified? [19:27:13] yes [19:27:23] +2 verified will result in a merge [19:27:37] i have to submit to merge [19:27:43] Change merged: Ottomata; [operations/software/varnish/varnishkafka] (master) - https://gerrit.wikimedia.org/r/70928 [19:27:52] but thar you go [19:28:03] awesome, thanks! [19:28:36] ty [19:29:56] oof, this merge was bad [19:29:57] hm [19:30:01] still trying [19:31:07] ottomata: I do not know how different the changes are on both branches, but would rebasing be easier? [19:31:17] !log bsitu Started syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master [19:31:27] Logged the message, Master [19:31:28] i tried rebasing too, but the changes are very different, buuuuut, i feel like that should be possible [19:31:48] qchris, basically, i deleted everything in github/master and started anew [19:31:49] ottomata: ok. May the merge be with you :-) [19:31:52] but the deletion was its own commit [19:31:58] so it should just be part of the history [19:32:10] <^d> Rewrite all the history! [19:32:13] there are some shared filenames though, so i think git is having fun with it [19:32:26] All your history belongs to us :-D [19:32:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [19:39:40] ^d: Did you see Saŝas comment for the gerrit/Firefox problem? [19:40:32] !log bsitu Finished syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master [19:40:41] Logged the message, Master [19:42:04] ergh, qchris, I got something pretty with rebase, but then I just realized that that won't work either, since that rewrites the history [19:42:15] and the commits are out there already [19:42:45] i'm starting to lean towards just doing this. anyone who has the old github master checked out will have to reclone or something [19:42:48] but meh [19:43:21] ottomata: Argh. You're of course right. [19:43:35] Ok, let's do it then .-) [19:43:36] <^d> qchris: No [19:43:49] haha [19:43:55] oh [19:43:57] that was to previous q [19:43:59] not to our convo [19:43:59] ok [19:44:10] <^d> qchris: Oh yes, I did. [19:44:21] ^d: I doubt it will buy us anything, but could we test it nonetheless, if I can find someone with Firefox problems? [19:44:33] <^d> Yes, let's try [19:45:07] ottomata: That was meant to our conversation as well. So, if you're good to merge the replication change. Just go ahead. [19:45:11] <^d> I still can't replicate with Firefox :\ [19:45:23] ^d: I could once ... [19:45:50] * qchris ponders whether something around here ran into the "Working ..."-stuck problem with firefox? [19:46:08] <^d> RoanKattouw was able to hit it pretty reliably I think. [19:46:25] ok, doing so [19:46:27] New review: Ottomata; "v0.1 has been synced. " [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/71248 [19:46:31] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71248 [19:46:31] s/something/someone/ [19:46:55] ^d: But never in dbg=1 mode [19:47:07] RoanKattouw: That's ok. [19:47:07] ok, cool, merged, running puppet on manganese [19:47:26] RoanKattouw: It will help nonetheless [19:47:35] <^d> RoanKattouw: What I need is your IP address so I can attempt to find what's going on in gerrit for you as it's working... [19:48:04] 216.38.130.164 , much good it'll do ya [19:48:54] qchris: woo, applied on manganese, how long do I have to wait to see what happens to github? [19:49:27] <^d> RoanKattouw: Ugh, you're right...office IP :\ [19:49:53] ottomata: Not sure about the replication plugin ... I think we'll have to restart the plugin to have it grab the new config. [19:50:09] <^d> Will do that, sec. [19:50:25] ottomata: I'll ask the gerrit overlord to do that afterwards. [19:50:47] ottomata: Oh. He's doing it already :-) [19:50:47] <^d> I just did that [19:51:24] <^d> RoanKattouw: Thanks anyway. I need someone on a unique IP. Not going to get anywhere with the office IP [19:52:01] I guess Nemo_bis was also affected by the Firefox "Working..."-stuck issue? [19:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:14] ok it feels like xmas eve! when will the replication ferry visit me?! [19:53:16] qchris: yes [19:53:19] <^d> qchris: [2013-07-02 19:51:21,053] INFO com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130701_1456_6951537312055118249.jar [19:53:34] and siebrand too I think [19:53:41] ^d: That's what I thought [19:53:43] ottomata: :-) The replication ferry will take a few minutes :-) [19:54:09] Nemo_bis: We're trying to debug the problem further. Are you on a unique IP address? [19:54:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [19:54:39] Is https://commons.wikimedia.org/wiki/Commons:Mus%C3%A9e_des_Augustins/test2 tracked in Bugzilla? [19:54:39] New review: Hashar; "Congratulations to both of you! \O/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53714 [19:54:47] Uncaught MWException [19:55:21] Has been around for at least a day or so. [19:55:35] qchris: yes, only this apartment should be using it; now opening some random patches, should I report those failing? [19:55:56] <^d> No, I just need your IP address, going to try and dive into the JVM and see what it's doing while "Working..." [19:56:02] Nemo_bis: If you find a single one that is failing for you, that'll do. [19:56:30] and of course now they all load [19:56:43] We fixed the problem! :-) [19:57:11] superm401: you can grep your exception code in the production exception.log [19:57:15] IP pm'ed, https://gerrit.wikimedia.org/r/#/c/71572/ is failing for me [19:57:24] superm401: should gives a hint about the error since there will be a stacktrace [19:57:34] superm401: that in turns let you find in bugzilla the issue (potenitally) [19:58:07] <^d> Hmm, so got a couple of connections here, some in ESTABLISHED, some in TIME_WAIT [19:58:17] <^d> All in TIME_WAIT now [19:58:56] superm401: "Invalid language code" :-] [19:59:11] hm, vanadium seems to have fallen off ganglia [19:59:30] hashar, will file [19:59:40] <^d> 24487: Unable to open socket file: target process not responding or HotSpot VM not loaded [19:59:40] <^d> The -F option can be used when the target process is not responding [19:59:48] <^d> qchris: ^ [20:00:02] superm401: I guess you can fill it against Translate which is not handling the exception [20:00:14] ^d: Maybe those were from Nemo_bis trying all those patches? [20:00:26] some where duplicates [20:00:26] hashar, looks like core to me, no? [20:00:29] <^d> Ah, -F, there we go. [20:00:30] *were [20:00:32] <^d> Silly jstack [20:00:45] ori-l: there was an increase of fatals at 6 UTC last time I saw it [20:01:03] Nemo_bis: yeah, trying to figure out what is happening [20:01:21] LeslieCarr: if I want to blast a ganglia metric away so that it picks up a new slope setting [20:01:27] can I just remove the .rrd file for that metric? [20:01:34] <^d> This output is basically useless. [20:01:43] :-(( [20:01:48] hashar, never mind, you're right. [20:02:18] Nemo_bis: Thanks for helping :-) [20:02:31] ^d: Anything that we can refer upstream to? [20:02:33] Well, I'm not sure what the root problem is but Translate is in the exception stack. [20:02:36] superm401: but the root cause is some wikitext markup passing an invalid language code [[:#if:]] , probably a typo in some template. Hard to track down though. [20:03:05] ottomata: have you done anything that would effect vanadium's ganglia graphs? [20:03:13] superm401: you can paste the stacktrace in the bug report. Double check it is not containing private data though :-] [20:03:18] ori-l: don't think so [20:03:35] <^d> Nemo_bis: Do you have a broken change open right now? [20:03:36] i haven't done anythingn with ganglia today except send a modified kafka metric to ganglia [20:03:44] which ahsn't been picked up by ganglia yet [20:03:59] ori [20:04:00] http://ganglia.wikimedia.org/latest/?c=Miscellaneous%20eqiad&h=vanadium.eqiad.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [20:04:05] ori-l: ^ [20:04:24] i've noticed sometimes that machines drop off from overview, dunno why [20:04:29] but they've always been there in the search [20:05:10] yeah, it's very strange. i was able to find it again by going to misc eqiad and selecting it from the dropdown, but trying to search for it by name doesn't work, and all existing URLs to graphs are blank [20:05:18] <^d> qchris: https://gerrit.wikimedia.org/jstack.out - meh. only thing mildly interesting is thread 18343. [20:05:20] maybe there's a token in the URL that expires eventually [20:05:54] wait, they're back now. [20:06:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:06:25] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:06:30] ^d: Does not look too good :-/ [20:06:35] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [20:06:55] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:05] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:05] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [20:07:15] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [20:07:47] Nemo_bis: graphs are back. dunno what that was about. maybe some disconnect between rrd and the front-end [20:07:55] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 20:07:50 UTC 2013 [20:07:55] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 20:07:50 UTC 2013 [20:07:55] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 20:07:50 UTC 2013 [20:08:05] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [20:08:15] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 20:08:11 UTC 2013 [20:08:15] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 20:08:12 UTC 2013 [20:08:15] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 20:08:14 UTC 2013 [20:08:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:25] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:08:25] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 20:08:18 UTC 2013 [20:08:35] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [20:08:45] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 20:08:39 UTC 2013 [20:08:45] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 20:08:40 UTC 2013 [20:08:45] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 20:08:41 UTC 2013 [20:08:55] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:05] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:05] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [20:09:05] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 20:08:58 UTC 2013 [20:09:05] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 20:08:59 UTC 2013 [20:09:05] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 20:09:03 UTC 2013 [20:09:15] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [20:09:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:25] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:09:25] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 20:09:23 UTC 2013 [20:09:25] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 20:09:23 UTC 2013 [20:09:35] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [20:09:45] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 20:09:40 UTC 2013 [20:09:45] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 20:09:43 UTC 2013 [20:09:55] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:05] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:05] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [20:10:05] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 20:09:58 UTC 2013 [20:10:05] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 20:09:59 UTC 2013 [20:10:15] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [20:10:15] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 20:10:10 UTC 2013 [20:10:15] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 20:10:10 UTC 2013 [20:10:15] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 20:10:13 UTC 2013 [20:10:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:25] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:10:35] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [20:10:36] ^d: Do you think that thread relates to the change view request? (It's GitOverHttpServlet, and UploadPack) [20:10:55] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:56] <^d> I doubt it, which is why I said it was only mildly interesting. [20:11:05] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:05] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [20:11:15] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [20:14:41] Nemo_bis: https://commons.wikimedia.org/wiki/Commons:Mus%C3%A9e_des_Augustins/test2 [20:15:35] ori-l: so it was just superm401 refreshing that page since 8.00 CEST? :) [20:15:58] Nemo_bis, :) spagewmf pointed it out to me originally. [20:16:04] But he didn't know of a bug, so I filed it. [20:16:05] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Tue Jul 2 20:15:59 UTC 2013 [20:16:06] https://bugzilla.wikimedia.org/show_bug.cgi?id=50606 [20:16:35] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [20:17:25] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Tue Jul 2 20:17:20 UTC 2013 [20:18:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:19:45] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Tue Jul 2 20:19:38 UTC 2013 [20:20:05] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [20:21:09] ^d: I tried locally and that code path is not hit, when looking at changes :-( I'll report that back upstream nonetheless. [20:21:28] <^d> Don't bother. I'm not even sure the right thread's in that dump. [20:21:31] <^d> I might've timed it badly. [20:22:10] We'll see what they say. [20:22:19] ^d: The other thing is replication. [20:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:35] ^d: I do not see it replicate the cdh4 repo. [20:22:55] ^d: Could you try starting the replication by hand? [20:22:56] <^d> Lemme force replication of it and tail thelog [20:23:12] ^d: Thanks. You're my hero :-) [20:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [20:24:18] <^d> :( http://p.defau.lt/?CDCwpO9OozbSPZBoWx6m8g [20:24:54] What? [20:25:03] I'll check. Thanks. [20:25:44] <^d> Replicated fine to all the other destinations, it's just github that complained. [20:25:58] * qchris likes github :-) [20:28:55] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Tue Jul 2 20:28:48 UTC 2013 [20:28:55] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Tue Jul 2 20:28:48 UTC 2013 [20:29:05] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:29:55] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:30:45] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Tue Jul 2 20:30:39 UTC 2013 [20:31:25] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:32:05] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Tue Jul 2 20:31:55 UTC 2013 [20:32:15] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [20:36:06] * MaxSem scaps [20:36:32] you're the scap man [20:36:48] sceepeedeepeepo po pa po do po [20:38:35] :) [20:40:21] holy '90s reference batman! [20:40:56] hrmmmm. this wolf is cloakless [20:41:00] but hi! [20:44:26] New patchset: Dzahn; "add AfterForm to RT4 to (re-)enable Password Reset feature on RT" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71719 [20:44:52] cookie to mutante for fast fixes :) [20:45:19] !log maxsem Started syncing Wikimedia installation... : Weekly mobile deployment [20:45:28] Logged the message, Master [20:45:50] yay mutante (password reset) [20:46:49] ^d: Sorry to bother you again. Could you try starting the replication again for operations/puppet/cdh4? [20:47:44] New patchset: Dzahn; "add AfterForm to RT4 to (re-)enable Password Reset feature on RT" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71719 [20:48:26] ottomata: sorry, yes [20:48:28] <^d> qchris: Looks like it replicated ok [20:49:33] ^d: Really? Github is still giving me the old master. [20:50:32] <^d> No errors on gerrit side. [20:50:44] :-) Ok. Thanks. [20:50:54] Maybe I am just too impatient. [20:56:03] !log maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment [20:56:12] Logged the message, Master [21:00:57] New patchset: Dzahn; "add AfterForm to RT4 to (re-)enable Password Reset feature on RT" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71719 [21:02:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:03:06] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [21:03:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [21:06:53] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [21:07:13] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [21:07:13] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [21:07:23] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [21:07:33] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [21:07:43] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [21:07:43] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [21:10:43] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [21:13:37] OK, I'm about to add a bunch of packages to brewster that I build using gem2deb. Is there some place I can document where these packages came from? (apart from inside the packages themselves, which is not necessarily easy thanks to the way gem2deb works) [21:14:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:44] ryan_lane, ^? [21:16:05] andrewbogott: in the packages themselves = i would see it if i do "apt-cache show " on some box later? [21:16:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.210 second response time [21:16:36] if it's in that Description: field there, sounds good [21:16:44] mutante, that sounds right… just, gem2deb makes the packages in one swoop. I gues I can dismantle them and then rebuild them. [21:18:36] andrewbogott: hmm, sounds like gem fetch + gem2tgz + dh-make-ruby + dpkg-buildpackage .. [21:19:04] of course more annoying than just one swoop.. [21:20:26] And there are 10 packages :( [21:20:49] are you gonna use them in puppet manifests? comment there? [21:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [21:46:11] Tim-away: AaronSchulz: RoanKattouw: https://bugzilla.wikimedia.org/show_bug.cgi?id=5382#c18 [21:46:31] Summary: Purge doesn't update link tables and page properties. Save does. [21:46:37] Reason is presumably performance. [21:47:24] It has previously been off our radar mostly, but there's starting to be more visible problems. Lately TemplateData. This block is usually added to the /doc page and (ironically) for performance reasons [21:47:35] so that we only have to purge /doc and not the Template and all its usage. [21:48:22] But because of this, the page properties adds to the host page aren't actually added to the Template page itself because our purge jobs discard the new page property values. [21:49:06] Can we fix the purge job to save those to the database? Note, this will not increase the number of purges. We're not talking about purging usage of templates, merely about the purge of pages as a result of another edit to not exclude link/page-prop updates anymore. [21:49:09] [21:51:13] basically getting rid of forcelinksupdate and making it the default (both in the API action=purge, in the UI action=purge and (most importantly) in the purge that is done from the job queue) [21:52:00] More comment 18/19: https://bugzilla.wikimedia.org/show_bug.cgi?id=5382#c18 [21:52:17] might want to ping Aaron|home instead :) [21:52:33] Aaron|home: ^ [22:00:02] MatmaRex: Looks like LinksUpdate does to page props and stuff by default. that makes sense actually, since it does update on the /doc page (e.g. api templatedata titels=tempalte foo/doc) [22:00:10] It's the ones that are queued for transcluding pages. [22:01:03] Krinkle: don't ask me how that works [22:01:14] if i had the slightest idea, i'd have fixed that myself :) [22:02:31] :) [22:02:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:34] so yeah, James_F and I just chatted about this issue generally. Sounds... ugh. So, if both Aaron|home and Tim-away can review the change and I guess getting paravoid involved as it implicated jobqueue resources... Then you can push out later today. I only say this because it seems like a "either this fix or we turn off VE" [22:03:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [22:07:22] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [22:07:32] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [22:07:32] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [22:07:52] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [22:08:02] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [22:08:02] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [22:08:02] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [22:08:30] Krinkle: thanks for the try Jquery email. it was useful. I'm looking forward to the more advanced versions [22:11:24] Krinkle: also, on https://gerrit.wikimedia.org/r/#/c/60783/19/modules/ext.openstack.Instance.js,unified line 95. If I use done, will I also need to make a fail to handle deferred? [22:11:45] I still need to find a good tutorial or set of documentation on deferred :( [22:12:05] RoanKattouw: I'm trying to figure out in the code what makes the initial update triggered for the origin page do include link tables and page properties, but the ones queued recursively for transcluding host pages not include. [22:12:12] .. that stuff [22:12:24] They seem exactly the same. [22:12:42] I was expecting some kind of mDoIncludeLinkUpdates=false/true; or something [22:12:55] especially confusing that all this is still called "LinkUpdate". [22:13:00] ah. right. you guys are likely knee deep in VE right now :) [22:13:52] Ryan_Lane: Thanks, note that the try jquery stuff goes pretty deep though. There's like over a dozen root chapters each with many sub sections and videos. But yeah, the first chapter is the most importnant. [22:14:11] And our stuff was supposed to be shot in amsterdam, but we didn't get to it. [22:14:21] yeah. I finished all of the tryJquery stuff available so far [22:14:29] Perhaps next time the 3 of us meet somewhere we can record that advanced meterial. [22:14:31] and signed up for the next chapter [22:14:36] Cool [22:14:55] Wait, you're saying the other chapters aren't available yet? That's too bad, I figured they'd be available by now. [22:15:23] it goes up to 5. Styling, right now [22:15:47] which is taming css and animation [22:19:16] James_F: greg-g: Aaron|home: AaronSchulz: Tim-away: RoanKattouw: Hm.. from looking at the code it seems like it is indeed running all secondary data updates. So they're not limited to just purging parser/html. In fact that doesn't happen at all from there, that's done on-demand when someone visits the page. [22:19:44] So back to square 1? What's causing these things not to be updated. [22:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.140 second response time [22:31:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [22:40:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [22:42:15] Krinkle: "on demand when someone visits the page"? What if no one visits the page prior to TemplateData being fetched? [22:42:52] RoanKattouw: I mean the html output of a page is not generated until someone visits the page afaik. We don't preparse every page that transcludes a page that is edited [22:43:01] Assuming that's true, that makes sense. [22:43:19] However we don't do the link updates on demand, otherwise we wouldn't have had this bug. [22:43:28] Because I have visited the page and see the new html [22:43:52] and if it takes too long I can manually purge it (no forced links update) to see it. [22:44:11] The problem is in the links update not happening, and I'm not sure why because the class is called "LinksUpdate" [22:44:52] if it doesnt' update the links, then what does do it for all those pages? The links table updates and other secondaty data updates (such as page properties) apparently dn't happen [22:45:00] and html doesn't happen until it is requested by a user [22:45:03] so what does it do? [22:45:06] Krinkle: is this for TemplateData? [22:45:08] RefreshJobs2 [22:45:35] ah, k [22:45:38] !log bsitu synchronized php-1.22wmf9/extensions/Echo 'update Echo to master' [22:45:45] gwicke: It is for a 1000 different things, but it includes TemplateData yes. When someone adds or updates on a Template:Foo/doc pages, apparently Template:Foo doesn't get it. [22:45:47] Logged the message, Master [22:46:25] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [22:46:25] So this isn't about pages using {{Foo}} being updated when Foo is edited but about Foo being updated when Foo/doc is edited. [22:46:30] !log bsitu synchronized php-1.22wmf8/extensions/Echo 'update Echo to master' [22:46:38] same principle though [22:46:38] Logged the message, Master [22:46:45] https://bugzilla.wikimedia.org/show_bug.cgi?id=5382 [22:47:26] WikiPage::doEditUpdates is called, getSecondaryDataUpdates is called, DataUpdate::runUpdates is called. [22:48:17] lemme share something that just took me a bit. if on a Debian/Ubuntu, you simply apt-get install puppet and then want to use "puppet apply foo.pp" to test stuff locally without a master, not even puppetmaster::self, and you wonder why many things work but some things don't like "unknown function keys", it's because they are part of the puppet module stdlib, and you need puppet module install puppetlabs/stdlib --modulepath=/etc/puppet/modul [22:49:16] < _rc> mutante: when you come to finalising your module remember to state that dependency in your modulefile [22:49:28] yea, stdlib needs to be a dependency too [23:01:05] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [23:06:05] PROBLEM - Puppet freshness on cp1027 is CRITICAL: No successful Puppet run in the last 10 hours [23:06:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [23:06:15] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [23:06:35] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [23:06:35] PROBLEM - Puppet freshness on niobium is CRITICAL: No successful Puppet run in the last 10 hours [23:06:45] PROBLEM - Puppet freshness on palladium is CRITICAL: No successful Puppet run in the last 10 hours [23:06:55] PROBLEM - Puppet freshness on strontium is CRITICAL: No successful Puppet run in the last 10 hours [23:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [23:36:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [23:47:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [23:52:54] RAWR. my vagrant somehow wiped out my vm again :( [23:53:46] well, it's still there, it seems [23:53:58] this is incredibly annoying [23:56:14] New patchset: Dzahn; "include exim::aliases::private on mchenry to puppetize mchenry alias files (class and files in private repo)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71743 [23:56:59] wth, that went wrong [23:58:05] ori-l: https://github.com/mitchellh/vagrant/issues/1755 [23:58:05] <^d> Ryan_Lane: Cool if I reboot manganese? [23:58:13] ^d: fine by me [23:58:30] ori-l: seems the id can't have a newline [23:58:53] which is of course a really great "wtf vagrant" issue :) [23:58:54] <^d> !log rebooting manganese (gerrit) [23:59:04] Logged the message, Master