[00:00:28] officeit-bgp , heh [00:01:04] cajoel: nice [00:01:38] jeremyb: wanna review https://gerrit.wikimedia.org/r/#/c/96915/ ? [00:01:45] jeremyb: you got all the reply about dickson, right [00:03:18] (03PS1) 10Cmjohnson: Removing mgmt for sq44 & sq48 [operations/dns] - 10https://gerrit.wikimedia.org/r/96917 [00:04:33] (03CR) 10MZMcBride: "I think I'd personally prefer to hold off on deploying this to private wikis until it's been stress-tested a bit more, but I trust you two" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 (owner: 10CSteipp) [00:04:49] (03CR) 10Yurik: [C: 04-2] "fixed by another patch, this will break it :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96824 (owner: 10Dr0ptp4kt) [00:05:41] (03PS2) 10Cmjohnson: Removing all dns for sq44 & sq48 [operations/dns] - 10https://gerrit.wikimedia.org/r/96917 [00:05:58] (03CR) 10Aaron Schulz: [C: 031] Hack: cron job to clean up tifs from /tmp on app servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/96915 (owner: 10Ori.livneh) [00:06:07] thanks AaronSchulz [00:06:29] (03CR) 10Cmjohnson: [C: 032] Removing all dns for sq44 & sq48 [operations/dns] - 10https://gerrit.wikimedia.org/r/96917 (owner: 10Cmjohnson) [00:06:39] (03CR) 10Ori.livneh: [C: 032] Hack: cron job to clean up tifs from /tmp on app servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/96915 (owner: 10Ori.livneh) [00:06:40] I thought you needed to escape \, but I was misreading. [00:06:54] (03CR) 10Chad: [C: 032] Increase Cirrus pool counter for new servers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96064 (owner: 10Manybubbles) [00:07:34] (03Merged) 10jenkins-bot: Increase Cirrus pool counter for new servers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96064 (owner: 10Manybubbles) [00:07:44] !log Built texvc for 1.23wmf5 [00:08:00] Logged the message, Master [00:08:36] !log demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'New pool counter settings for Cirrus' [00:08:52] Logged the message, Master [00:09:07] !log demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'New pool counter settings for Cirrus' [00:09:21] Logged the message, Master [00:14:31] (03CR) 10Dzahn: [C: 032] fix and remove various planet feed URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/96914 (owner: 10Dzahn) [00:15:35] mutante: no, busy, didn't read it yet; :) /me runs off again [00:17:40] (03CR) 10Dzahn: [C: 032] etherpad - tabbing, quoting & aligning [operations/puppet] - 10https://gerrit.wikimedia.org/r/96354 (owner: 10Dzahn) [00:22:45] (03CR) 10Dzahn: [C: 032] retab, quoting, linting of ishmael.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/96362 (owner: 10Dzahn) [00:28:51] (03PS7) 10Mwalker: Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [00:31:44] (03CR) 10Dzahn: "the dependency has been merged meanwhile. still good to go?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96424 (owner: 10Dzahn) [00:32:57] ^d is your stuff on tin already? [00:33:05] I'm about to scap [00:33:07] <^d> I already sync'd it. [00:33:14] cool :) [00:33:14] <^d> Fire away [00:35:21] !log ori synchronized php-1.23wmf4/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebInfobox.js 'Cherry-pick I3efc1fa64' [00:35:36] Logged the message, Master [00:36:20] !log csteipp started scap: Updating WikimediaMessages to master for OAuth message [00:36:36] Logged the message, Master [00:36:47] yay. i'm not going to miss the ellipsis. [00:37:04] :) [00:39:18] Hmm. "mw25: rsync: send_files failed to open "/php-1.23wmf5/includes/Sanitizer.php.save"" [00:40:15] csteipp: !log ? [00:44:49] (03PS1) 10Cmjohnson: Removing DNS entries for ms7 and ms8 [operations/dns] - 10https://gerrit.wikimedia.org/r/96927 [00:44:50] going to touch and sync wmf4's startup.js; it won't interfere with scap [00:45:13] Hey Reedy, want to remove your temp file :) [00:46:13] !log ori synchronized php-1.23wmf4/resources/startup.js 'touch' [00:46:17] ty git am [00:46:28] Logged the message, Master [00:50:21] !log ori synchronized php-1.23wmf4/extensions/MobileFrontend/javascripts/loggingSchemas/MobileWebInfobox.js 'Cherry-pick I3efc1fa64' [00:50:26] (03CR) 10Cmjohnson: [C: 032] Removing DNS entries for ms7 and ms8 [operations/dns] - 10https://gerrit.wikimedia.org/r/96927 (owner: 10Cmjohnson) [00:50:29] !log csteipp finished scap: Updating WikimediaMessages to master for OAuth message [00:50:36] Logged the message, Master [00:50:44] csteipp: did you get timing info? [00:50:48] in stdout? [00:50:51] Logged the message, Master [00:50:58] ori-l: scap completed in 16m 09s. [00:51:04] That's amazing [00:51:15] (compared with previous) [00:51:32] :) [00:52:01] cmprss all the things [00:52:20] --go-faster [00:53:19] <^d> Who needs a new deployment tool? We'll just let ori-l make us `faster-scap` ;-) [00:53:25] is it like -v in lspci? the more -f (--faster) you put, the faster it goes? -fffffffffffffff [00:53:55] i was actually asking just to make sure the timing is outputted correctly [00:54:12] <^d> greg-g: Not quite. [00:54:21] <^d> --fffffuuuuuuuu [00:54:53] ^d: that's the one that crashes all mwXXXX's, right? [00:55:33] <^d> Yep. [00:56:25] !log ori synchronized php-1.23wmf4/resources/startup.js 'touch' [00:56:39] Logged the message, Master [00:58:11] (03PS4) 10CSteipp: Enable OAuth on all public wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 [00:59:54] (03CR) 10CSteipp: [C: 032] Enable OAuth on all public wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 (owner: 10CSteipp) [01:00:05] (03Merged) 10jenkins-bot: Enable OAuth on all public wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96890 (owner: 10CSteipp) [01:09:30] Dangit. You know, I loved scap so much, I'm going to do it again.. [01:11:36] scap: it runs twice as fast, *twice as often* [01:11:39] that's 4x! [01:15:39] (03PS1) 10Legoktm: Undeploy AssertEdit (merged into core) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/96931 [01:16:04] !log csteipp started scap: Really send the new messages this time.. [01:16:21] Logged the message, Master [01:23:02] Ryan_Lane, paravoid: if I have a package in a PPA (it's a backport of texlive 2012 into ubuntu 12.04) -- how does that work with puppetization? can we import it into our local repos? [01:24:16] mwalker: i think that has been done? i guess partly would depend on where it's from (anyone can make a ppa!) [01:24:37] it's from the maintainers of texlie [01:24:40] *texlive [01:37:11] !log csteipp finished scap: Really send the new messages this time.. [01:37:27] Logged the message, Master [01:38:04] (03CR) 10Dzahn: "one inline comment and fyi the patch it depended on has been merged" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn) [01:40:42] !log csteipp synchronized wmf-config/CommonSettings.php [01:40:57] Logged the message, Master [01:41:11] !log csteipp synchronized wmf-config/InitialiseSettings.php [01:41:26] Logged the message, Master [01:45:36] (03CR) 10Dzahn: [C: 04-1] "thanks, it's nice that you are starting as a module right away, please see inline comments though and add some more reviewers to talk abou" (038 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 (owner: 10Addshore) [01:46:32] greg-g: Deployment is done ^ Sorry it took so long [01:48:00] (03PS2) 10Dzahn: qualify vars planet_domain_name, planet_languages [operations/puppet] - 10https://gerrit.wikimedia.org/r/96225 (owner: 10ArielGlenn) [01:50:35] (03PS2) 10Dzahn: role classes for download servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/96415 [01:55:56] (03CR) 10Dzahn: [C: 031] "this should be fine, it's more a matter of taste, merge if Ariel likes it since Ariel is kind of the owner of this role" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96415 (owner: 10Dzahn) [02:09:00] (03CR) 10Dzahn: [C: 032] qualify vars planet_domain_name, planet_languages [operations/puppet] - 10https://gerrit.wikimedia.org/r/96225 (owner: 10ArielGlenn) [02:16:36] !log LocalisationUpdate completed (1.23wmf4) at Fri Nov 22 02:16:36 UTC 2013 [02:16:53] Logged the message, Master [02:21:06] (03CR) 10Dzahn: "thanks Ariel, no issues at all with this" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96225 (owner: 10ArielGlenn) [02:32:21] (03PS1) 10Dzahn: more broken planet feeds [operations/puppet] - 10https://gerrit.wikimedia.org/r/96935 [02:33:13] (03CR) 10Dzahn: [C: 031] "these things are handled on https://meta.wikimedia.org/wiki/Planet_Wikimedia" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96935 (owner: 10Dzahn) [02:35:52] !log LocalisationUpdate completed (1.23wmf5) at Fri Nov 22 02:35:51 UTC 2013 [02:36:08] Logged the message, Master [02:48:54] (03PS1) 10Dzahn: fix moved feed URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/96938 [02:50:04] (03PS2) 10Dzahn: fix moved feed URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/96938 [02:50:45] (03CR) 10Dzahn: [C: 032] fix moved feed URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/96938 (owner: 10Dzahn) [03:23:34] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 22 03:23:33 UTC 2013 [03:23:50] Logged the message, Master [03:41:02] (03PS1) 10Tim Starling: Normalise the path part of URLs in the text frontend [operations/puppet] - 10https://gerrit.wikimedia.org/r/96941 [03:41:43] (03CR) 10Dzahn: "maybe, i'd like to keep possible changes to the reporter scripts separate from this specific change though and hear andre if we want to ac" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [03:53:42] (03CR) 10Dzahn: "Ariel, re: install_certificate{ $svc_name: } in apache.pp etc. no, actually don't expect those certs to be moved into the module, at least" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [04:49:52] (03PS2) 10Springle: remove bellin/blondel references, they don't exist [operations/puppet] - 10https://gerrit.wikimedia.org/r/92989 (owner: 10Dzahn) [04:52:47] (03CR) 10Springle: [C: 032] remove bellin/blondel references, they don't exist [operations/puppet] - 10https://gerrit.wikimedia.org/r/92989 (owner: 10Dzahn) [05:33:38] (03PS1) 10Ori.livneh: graphite::web: parametrize site_name; declare in role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/96952 [06:28:27] (03CR) 10ArielGlenn: "OK, it makes sense to leave certs where they are for now. But at some point we should have certs live in the (role) modules where they ar" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [06:53:09] !log neon back to read-only root filesystem, investigating [06:53:22] Logged the message, Master [06:54:44] (03CR) 10Ori.livneh: [C: 032] graphite::web: parametrize site_name; declare in role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/96952 (owner: 10Ori.livneh) [06:58:33] (03PS2) 10Mattflaschen: Remove "Your cache administrator is nobody" joke. [operations/puppet] - 10https://gerrit.wikimedia.org/r/95147 [06:59:53] (03PS1) 10Ori.livneh: graphite::web: Correct parameter name [operations/puppet] - 10https://gerrit.wikimedia.org/r/96959 [07:00:52] (03CR) 10Ori.livneh: [C: 032] graphite::web: Correct parameter name [operations/puppet] - 10https://gerrit.wikimedia.org/r/96959 (owner: 10Ori.livneh) [07:03:27] (03PS1) 10Ori.livneh: rewrite nginx module [operations/puppet] - 10https://gerrit.wikimedia.org/r/96961 [07:06:46] ! rebooting neon, no underlying errors found except for the initial ext3 'deleted inode reference' which caused / to be remounted r/o [07:06:54] !log rebooting neon, no underlying errors found except for the initial ext3 'deleted inode reference' which caused / to be remounted r/o [07:07:09] Logged the message, Master [07:07:37] (03PS8) 10Mwalker: Initial Puppet Try for OCG::Collection Role [operations/puppet] - 10https://gerrit.wikimedia.org/r/96811 [07:08:05] * apergos sighs [07:14:29] well that worked but I sure wish I knew what is actually broken, because otherwise we'll have this again in not too long [07:16:30] faulty RAM? [07:16:55] possible [07:17:13] bad sector on disk? [07:17:25] didn't see any disk-related errors at a lower level [07:17:42] first error was ext3 whining that a deleted inode was referenced [07:19:21] its a bit farfetched at ground level -- but I always love to blame things on high energy particles flipping bits [07:19:41] twice in two days... mmmm [07:19:58] that's a bit more unlikely though [07:20:44] elves? [07:20:51] communists? [07:20:57] pixies! [07:21:24] de-ba-ser [07:21:28] did any other application complain about not being able to access a file? [07:21:38] there could be something holding a handle open [07:21:47] and I would expect it to throw an error [07:22:39] no, I didn't see anything like that [07:22:45] I'm on another track right now, hold on [07:23:07] * mwalker watches for oncoming trains [07:25:21] are you implying that communists are as nonexistent as pixies? :-P [07:25:52] oh no; just that they're as likely a candidate for mysterious bit flipping [07:26:06] hahaha [07:29:08] ok so what's interesting about these two fs issues is they were both the same time of day and the *same inode* [07:29:21] Nov 22 06:33:22 neon kernel: [76964.339556] EXT3-fs error (device md0): ext3_lookup: deleted inode referenced: 21531 [07:29:23] today's [07:29:42] Nov 21 06:32:16 neon kernel: [1285011.010472] EXT3-fs error (device md0): ext3_lookup: deleted inode referenced: 21531 [07:29:45] yesterday's [07:34:03] well; if you wanted to wait a while; you could try to `find -printf "%i:\t%p\n" | egrep "^21531\t"` [07:34:23] aka; try and see if there's a file that matches that inode [07:34:45] I guess inum finds by inode [07:34:56] but I was going to see if there's some quicker way to get info about it [07:53:42] I have managed to trigger the error already by the find [07:54:27] /lib/modules/3.2.0-45-generic/kernel/net/netfilter/xt_hashlimit.ko this [07:54:52] there are four files that trigger these errors. I wonder what I can do about it [07:55:08] find: `/lib/modules/3.2.0-45-generic/kernel/net/netfilter/xt_hashlimit.ko': Input/output error [07:55:08] find: `/lib/modules/3.2.0-45-generic/kernel/net/netfilter/xt_u32.ko': Input/output error [07:55:08] find: `/lib/modules/3.2.0-45-generic/kernel/net/netfilter/xt_esp.ko': Input/output error [07:55:08] find: `/lib/modules/3.2.0-45-generic/kernel/net/netfilter/xt_socket.ko': Input/output error [07:55:16] the find completes after these. [07:55:25] recompile the netfilter module? [07:55:55] or actually; easier -- downgrade the kernel [07:56:02] 3.2.0-53-generic this is what we are running [07:56:14] I need to get rid of those somehow, but in theory they are already gone? [07:56:18] oh... [07:56:20] hmm [07:56:33] ext3_lookup: deleted inode referenced: 21525 and so on [07:56:37] for all four of them [07:56:58] in the meantime we are read only again on neon until the next reboot [08:00:57] can you touch and then delete the files? [08:01:42] I can't ls them so I would guess I can't touch them anything [08:01:48] s/anything/either/ [08:02:10] I mean any filesystem operation will involve referencing the inode which will cause barf -> r/o mode [08:02:27] I was wondering if touching it would create the inode [08:05:01] * mwalker wonders if you could breed them by hardlinking [08:05:14] * apergos shudders [08:08:12] I'm sorta scared to try; but ln has a -f option which 'removes existing destination files' [08:08:25] if it doesn't look first you might be able to use that [08:11:39] well for right now I'm going to reboot this so we're back in r/w with monitoring again [08:14:26] !log rebooting neon again, found four filenames with deleted inode but side effect is back in r/o for / [08:14:43] Logged the message, Master [08:28:44] (03CR) 10Odder: "Nameserver settings for wikimedia.pl were changed with bug 33509, so I guess it wouldn't be too hard to change them to ns*.wikimedia.org." [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn) [08:33:54] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 3 hours [08:33:54] PROBLEM - Puppet freshness on mc1003 is CRITICAL: No successful Puppet run in the last 3 hours [08:33:54] PROBLEM - Puppet freshness on mw16 is CRITICAL: No successful Puppet run in the last 3 hours [08:33:54] PROBLEM - Puppet freshness on pc2 is CRITICAL: No successful Puppet run in the last 3 hours [08:33:54] PROBLEM - Puppet freshness on sq67 is CRITICAL: No successful Puppet run in the last 3 hours [08:33:54] PROBLEM - Puppet freshness on stafford is CRITICAL: No successful Puppet run in the last 3 hours [08:34:08] ughh [08:34:54] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:54] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:54] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:54] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [08:34:54] PROBLEM - Puppet freshness on labsdb1002 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:54] PROBLEM - Puppet freshness on labsdb1003 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:54] PROBLEM - Puppet freshness on mc1015 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:55] PROBLEM - Puppet freshness on mw1018 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:55] PROBLEM - Puppet freshness on mw1088 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:56] PROBLEM - Puppet freshness on mw1099 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:56] PROBLEM - Puppet freshness on mw1100 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:57] PROBLEM - Puppet freshness on mw1103 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:57] PROBLEM - Puppet freshness on mw1117 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:58] PROBLEM - Puppet freshness on mw1164 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:58] PROBLEM - Puppet freshness on mw1176 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:59] PROBLEM - Puppet freshness on mw1217 is CRITICAL: No successful Puppet run in the last 3 hours [08:34:59] PROBLEM - Puppet freshness on mw51 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:00] PROBLEM - Puppet freshness on search1010 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:00] PROBLEM - Puppet freshness on sq71 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:01] PROBLEM - Puppet freshness on sq83 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:01] PROBLEM - Puppet freshness on sq86 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:02] PROBLEM - Puppet freshness on srv287 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:02] PROBLEM - Puppet freshness on srv300 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:46] (03CR) 10Odder: "All except stuartgeiger.com work for me, I'll contact Stuart on-wiki and ask him to fix his website." [operations/puppet] - 10https://gerrit.wikimedia.org/r/96935 (owner: 10Dzahn) [08:35:54] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:54] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:54] PROBLEM - Puppet freshness on cp1020 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:54] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:54] PROBLEM - Puppet freshness on cp1061 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:54] PROBLEM - Puppet freshness on cp3007 is CRITICAL: No successful Puppet run in the last 3 hours [08:35:54] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 3 hours [08:36:24] those are lies. [08:37:54] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 3 hours [08:37:54] PROBLEM - Puppet freshness on cp1006 is CRITICAL: No successful Puppet run in the last 3 hours [08:37:54] PROBLEM - Puppet freshness on cp1051 is CRITICAL: No successful Puppet run in the last 3 hours [08:37:54] PROBLEM - Puppet freshness on cp1069 is CRITICAL: No successful Puppet run in the last 3 hours [08:37:54] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:54] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:54] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:54] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:54] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:54] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 3 hours [08:38:55] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:55] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:56] PROBLEM - Puppet freshness on lvs4 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:56] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:57] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:57] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:58] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:58] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:58] (03CR) 10Odder: "All fine for me except the removal of wikiźródła.pl, which works for me; might this be related to the fact that it's an IDN domain and Ven" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96914 (owner: 10Dzahn) [08:38:59] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 3 hours [08:38:59] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:00] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:00] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:01] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:01] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:02] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:02] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:03] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [08:39:03] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:49] ugh [08:39:54] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:54] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:54] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:54] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 3 hours [08:39:54] PROBLEM - Puppet freshness on es6 is CRITICAL: No successful Puppet run in the last 3 hours [08:40:02] it thinks all the results for these are stale [08:41:54] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:54] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:54] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:54] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:54] PROBLEM - Puppet freshness on db63 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:54] PROBLEM - Puppet freshness on elastic1008 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:54] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 3 hours [08:41:55] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:55] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:56] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:56] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:57] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:57] PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:58] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 3 hours [08:41:58] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:59] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 3 hours [08:41:59] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 3 hours [08:42:00] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 3 hours [08:42:00] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 3 hours [08:42:54] Nov 22 08:40:44 neon icinga: Warning: The results of service 'Puppet freshness' on host 'snapshot1' are stale by 0d 0h 0m 54s (threshold=0d 3h 0m 0s). I'm forcing an immediate check of the service. how is 54 seconds past the threshhold?? [08:42:54] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 3 hours [08:42:54] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 3 hours [08:42:54] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 3 hours [08:42:54] PROBLEM - Puppet freshness on db48 is CRITICAL: No successful Puppet run in the last 3 hours [08:42:54] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:44:54] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 3 hours [08:44:54] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 3 hours [08:44:54] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 3 hours [08:44:54] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 3 hours [08:44:54] PROBLEM - Puppet freshness on db1021 is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on cp1018 is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:45:54] PROBLEM - Puppet freshness on db9 is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on capella is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 3 hours [08:46:54] PROBLEM - Puppet freshness on cp1016 is CRITICAL: No successful Puppet run in the last 3 hours [08:47:54] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 3 hours [08:47:54] PROBLEM - Puppet freshness on amssq31 is CRITICAL: No successful Puppet run in the last 3 hours [08:47:54] PROBLEM - Puppet freshness on amssq35 is CRITICAL: No successful Puppet run in the last 3 hours [08:47:54] PROBLEM - Puppet freshness on amssq37 is CRITICAL: No successful Puppet run in the last 3 hours [08:47:54] PROBLEM - Puppet freshness on cp1062 is CRITICAL: No successful Puppet run in the last 3 hours [08:48:54] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 3 hours [08:48:54] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 3 hours [08:48:54] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 3 hours [08:48:54] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 3 hours [08:48:54] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:54] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:54] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:54] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:54] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:54] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:55] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:55] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:56] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:56] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:57] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:57] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:58] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:58] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:59] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 3 hours [08:49:59] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:00] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:00] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:01] PROBLEM - Puppet freshness on ssl1007 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:01] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 3 hours [08:50:02] PROBLEM - Puppet freshness on testsearch1002 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:02] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on lvs1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:54] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:55] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:55] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:56] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:56] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:57] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:57] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:58] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:58] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:59] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 3 hours [08:50:59] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:00] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:00] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:01] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:01] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:02] PROBLEM - Puppet freshness on testsearch1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:02] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:03] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:27] still lies >_< [08:51:54] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:54] PROBLEM - Puppet freshness on amssq40 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:54] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:54] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 3 hours [08:51:54] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 3 hours [08:53:54] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 3 hours [08:53:54] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 3 hours [08:53:54] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 3 hours [08:53:54] PROBLEM - Puppet freshness on cp1053 is CRITICAL: No successful Puppet run in the last 3 hours [08:53:54] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 3 hours [08:54:54] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 3 hours [08:54:54] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 3 hours [08:54:54] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 3 hours [08:54:54] PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 3 hours [08:54:54] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on db1015 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on db1018 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 3 hours [08:55:54] PROBLEM - Puppet freshness on elastic1005 is CRITICAL: No successful Puppet run in the last 3 hours [08:56:54] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 3 hours [08:56:54] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 3 hours [08:56:54] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 3 hours [08:56:54] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 3 hours [08:56:54] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 3 hours [08:57:54] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 3 hours [08:57:54] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: No successful Puppet run in the last 3 hours [08:57:54] PROBLEM - Puppet freshness on arsenic is CRITICAL: No successful Puppet run in the last 3 hours [08:57:54] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 3 hours [08:57:54] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 3 hours [08:58:54] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 3 hours [08:58:54] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 3 hours [08:58:54] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 3 hours [08:58:54] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 3 hours [08:58:54] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:54] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:54] PROBLEM - Puppet freshness on cp1048 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:54] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:54] PROBLEM - Puppet freshness on db1030 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:54] PROBLEM - Puppet freshness on db1016 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:55] PROBLEM - Puppet freshness on hooft is CRITICAL: No successful Puppet run in the last 3 hours [08:59:55] PROBLEM - Puppet freshness on es8 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:56] PROBLEM - Puppet freshness on lvs2 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:56] PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:57] PROBLEM - Puppet freshness on ms1002 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:57] PROBLEM - Puppet freshness on mw1034 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:58] PROBLEM - Puppet freshness on mc1004 is CRITICAL: No successful Puppet run in the last 3 hours [08:59:58]