[00:00:04] RoanKattouw, ^d, marktraceur, MaxSem, RoanKattouw: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141127T0000). [00:00:14] * RoanKattouw is still submitting things to SWAT [00:00:24] do we even SWAT today? [00:00:40] WHO'S UP FOR FIXING SHIT TOMORROW? :P [00:01:47] Stupid mid week holidays [00:02:11] I'll do SWAT today [00:02:15] Given that most of it is mine/ours [00:02:17] YuviPanda: ok, updated phab [00:02:20] that didn't hurt much [00:02:26] i have to go offline for ~30min [00:02:27] cscott: :) [00:02:27] (03PS1) 10Yuvipanda: ocg: Temp hack to fix empty redis password, take 2 [puppet] - 10https://gerrit.wikimedia.org/r/176188 [00:02:31] send email if you need me urgently [00:02:31] cscott: Uhm... [00:02:37] cscott: You have a SWAT scheduled for right now [00:02:45] cscott: ok. if this patch doesn't fix it, I'll just disable puppet and mod the file by hadn [00:02:47] *hand [00:02:55] RoanKattouw: yeah, i'm pretty sure that patch already got merged earlier today. [00:02:58] cscott: If you're running away now I'm not going to deploy your Re-enable Download as PDF patch unless you delegate somebody [00:03:12] Oh, hah [00:03:39] RoanKattouw: sorry for the confusion! [00:04:00] OK yeah it's already deployed [00:04:29] cscott: ping [00:04:42] (03CR) 10Yuvipanda: [C: 032] ocg: Temp hack to fix empty redis password, take 2 [puppet] - 10https://gerrit.wikimedia.org/r/176188 (owner: 10Yuvipanda) [00:05:48] (03CR) 10Catrope: [C: 032] Followup 313c29f: correct spelling of wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176182 (owner: 10Catrope) [00:06:04] (03Merged) 10jenkins-bot: Followup 313c29f: correct spelling of wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176182 (owner: 10Catrope) [00:06:39] cscott_away: gwicke so ocg should be back up now. can you verify? [00:06:42] (03CR) 10GWicke: "One case that doesn't seem to be covered yet is this type of log entry:" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:06:58] YuviPanda: it should be [00:07:00] !log catrope Synchronized wmf-config/InitialiseSettings.php: Re-enable VisualEditor on frwiktionary and svwiktionary (duration: 00m 06s) [00:07:04] Logged the message, Master [00:07:11] gwicke: the redis password died again, I just hacked in a fix [00:07:38] YuviPanda: hmm, how did it 'die'? [00:07:48] gwicke: the password got reset to '' again by puppet [00:09:10] !log catrope Synchronized php-1.25wmf9/extensions/VisualEditor: SWAT (duration: 00m 05s) [00:09:12] Logged the message, Master [00:09:18] !log catrope Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 07s) [00:09:20] Logged the message, Master [00:09:45] (03PS1) 10Yuvipanda: Revert "ocg: Temp hack to fix empty redis password, take 2" [puppet] - 10https://gerrit.wikimedia.org/r/176190 [00:10:51] (03CR) 10Yuvipanda: [C: 032] Revert "ocg: Temp hack to fix empty redis password, take 2" [puppet] - 10https://gerrit.wikimedia.org/r/176190 (owner: 10Yuvipanda) [00:11:28] YuviPanda: thanks! [00:11:42] gwicke: still poking at it, though. [00:12:17] I am not aware of any changes in that area, so am wondering why it's failing now [00:12:41] gwicke: potentially a hiera issue, I'm unsure either. [00:12:53] gwicke: can you verify OCG still works, from a user perspective? [00:13:15] ahh, maybe it's related to https://gerrit.wikimedia.org/r/#/c/174694/ [00:13:38] checking.. [00:13:41] (03CR) 10CSteipp: "Gwicke, looks like it's missing" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:14:22] (03PS1) 10BryanDavis: Use hiera to configure udp2log endpoint for ::mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/176191 [00:14:24] (03CR) 10GWicke: "@CSteipp: yup, will amend scott's patch in a bit." [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:14:34] YuviPanda: works for me [00:14:39] gwicke: cool, thanks! [00:14:44] I'll email ops@ and then go to sleep [00:14:59] YuviPanda: okay, thanks for looking into it! [00:15:04] gwicke: yw! [00:16:16] (03PS3) 10GWicke: Further improvements to OCG apparmor profile. [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:16:31] csteipp: ^^ [00:17:23] csteipp: wait, forgot the xetex section [00:18:48] (03PS4) 10GWicke: Further improvements to OCG apparmor profile. [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:19:11] csteipp: now with xetex inside™ [00:20:45] (03CR) 10CSteipp: [C: 031] Further improvements to OCG apparmor profile. [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:22:46] (03CR) 10Dzahn: [C: 032] Further improvements to OCG apparmor profile. [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [00:24:06] (03PS2) 10Yuvipanda: Add hiera lookup tool [puppet] - 10https://gerrit.wikimedia.org/r/175153 (owner: 10Ori.livneh) [00:30:15] gwicke: this warning about ocg has been apparently around for days: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=ocg1002&service=OCG+health [00:30:16] !log catrope Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 06s) [00:30:18] Logged the message, Master [00:30:19] is that ok / known? [00:31:16] YuviPanda: not sure, but to me it sounds like some queue is possibly getting too long [00:31:24] hmm, ok [00:31:31] but you say user facing functionality is ok [00:31:34] and that's good enough, I guess [00:32:12] OCG has this elaborate system of queues, and I could imagine that one of those is not properly garbage collected, without affecting new jobs [00:32:49] gwicke: heh, ok. in that case that monitoring check should be fixed, I guess. also made more clear what exactly it is doing. and if there are multiple queues, each should be monitored... [00:32:51] I'll ask cscott & arlolra about it [00:32:58] alright [00:33:10] thanks for letting us know! [00:33:23] yw! [00:33:26] I'm going to file a bug anyway [00:34:52] cool, thanks! [00:34:56] mail sent too [00:35:19] gwicke: https://phabricator.wikimedia.org/T76115 done [00:36:51] YuviPanda: thanks! [00:36:57] yw! [00:37:06] I'll probably be up for another 5 mins, and then head to bed [00:39:48] (03PS2) 10BryanDavis: Use hiera to configure udp2log endpoint for ::mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/176191 [00:44:42] YuviPanda: thanks & enjoy your sleep! [00:44:48] will do! [00:56:11] back. thanks again, YuviPanda! [00:56:24] cscott: :) I filed another bug about the icinga warnings [00:56:32] (the red sox recently acquired a panda. obligatory off-topic aside) [00:56:38] hahaha :) [00:56:43] YuviPanda: yeah, i don't think they are anything to worry about. [00:56:46] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 1 failures [00:56:55] yeah, in which case they shouldn't be in warning state :) [00:56:56] they seem to date to the time when we increased the cache lifetime from 2 days to 4. [00:57:06] so we should fix the monitoring [00:57:11] right, we forgot to increase the warning level at that time [00:57:24] so, definitely something to fix, but nothing to lose sleep over [00:57:31] cscott: can you mention that in the bug as well? [00:57:39] yeah, will do [00:57:46] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 1 failures [00:57:53] hmm, what now icinga-wm [01:00:04] Dear anthropoid, the time has come. Please deploy US Holiday (Thanksgiving) - New Deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141127T0100). [01:00:12] hahahahaha [01:00:16] well done, jouncebot [01:00:19] well done [01:00:27] did it say New Deploys or No deploys? [01:00:53] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:00:58] (03CR) 10Cscott: "That seems like that will work. I just don't understand why it's needed; the move to postmortem should happen after xelatex ends. It prob" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [01:01:11] !lоg hoo synchronized /thanksgiving [01:01:35] hmm, virtxxxx failures were just transient [01:02:51] PROBLEM - HHVM busy threads on mw1229 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [01:05:18] PROBLEM - HHVM busy threads on mw1221 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [115.2] [01:05:18] PROBLEM - HHVM busy threads on mw1226 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:05:41] PROBLEM - HHVM busy threads on mw1225 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:05:44] (03CR) 10BryanDavis: "Tested in beta via cherry-pick. This has restored rsyslog forwarding of messages from HHVM to the udp2log agent on deployment-bastion (T74" [puppet] - 10https://gerrit.wikimedia.org/r/176191 (owner: 10BryanDavis) [01:05:46] PROBLEM - HHVM busy threads on mw1230 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:06:37] bd808: I wonder if you should split out the backend order switching into a separate patch [01:07:03] (03CR) 10BryanDavis: "See associated hiera config at " [puppet] - 10https://gerrit.wikimedia.org/r/176191 (owner: 10BryanDavis) [01:07:12] (03CR) 10GWicke: "Could there be an overlap between those tools running & the move? In that case I could imagine the files still being open & written to whi" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [01:07:26] YuviPanda: I certainly could [01:07:33] bd808: that would be nice, I think :) [01:07:38] please do! [01:07:46] will do then [01:07:59] bd808: ty! [01:08:21] RECOVERY - HHVM busy threads on mw1225 is OK: OK: Less than 1.00% above the threshold [76.8] [01:08:51] (03PS1) 10GWicke: Next round of OCG apparmor fixes [puppet] - 10https://gerrit.wikimedia.org/r/176202 [01:09:33] (03CR) 10GWicke: "The saga continues: https://gerrit.wikimedia.org/r/176202" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [01:10:32] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:10:32] RECOVERY - HHVM busy threads on mw1221 is OK: OK: Less than 1.00% above the threshold [76.8] [01:10:32] RECOVERY - HHVM busy threads on mw1226 is OK: OK: Less than 1.00% above the threshold [76.8] [01:10:52] RECOVERY - HHVM busy threads on mw1229 is OK: OK: Less than 1.00% above the threshold [76.8] [01:11:01] RECOVERY - HHVM busy threads on mw1230 is OK: OK: Less than 1.00% above the threshold [76.8] [01:11:03] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 1 failures [01:13:41] (03PS3) 10BryanDavis: Use hiera to configure udp2log endpoint for ::mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/176191 [01:13:43] (03PS1) 10BryanDavis: labs: make mwyaml hiera backend first lookup [puppet] - 10https://gerrit.wikimedia.org/r/176204 [01:14:49] (03CR) 10BryanDavis: "Split hiera backend order change into Ib718cc8 at Yuvi's request." [puppet] - 10https://gerrit.wikimedia.org/r/176191 (owner: 10BryanDavis) [01:17:04] (03CR) 10BryanDavis: "Without this change, hiera would choose a value found in the hieradata/**/*.yaml files over a setting made on wikitech. This seemed very u" [puppet] - 10https://gerrit.wikimedia.org/r/176204 (owner: 10BryanDavis) [01:17:25] (03CR) 10GWicke: "@CScott: Actually, I think the file_inherit bit is about the xetex child process inheriting the open file descriptor from the parent (node" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [01:19:30] (03CR) 10GWicke: "Fortunately, fixed in 0.11: https://github.com/joyent/node/issues/6905" [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [01:22:05] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [01:25:51] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 1 failures [01:31:16] (03CR) 10Tim Starling: [C: 031] "As long as no files are owned by nobody." [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [01:35:22] (03CR) 10Cscott: [C: 04-1] Next round of OCG apparmor fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/176202 (owner: 10GWicke) [01:35:42] (03CR) 10Cscott: "ok, makes sense." [puppet] - 10https://gerrit.wikimedia.org/r/176155 (owner: 10Cscott) [01:39:30] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:03:33] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: puppet fail [02:11:26] (03CR) 10GWicke: Next round of OCG apparmor fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/176202 (owner: 10GWicke) [02:11:31] (03PS2) 10GWicke: Next round of OCG apparmor fixes [puppet] - 10https://gerrit.wikimedia.org/r/176202 [02:15:34] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 1 failures [02:15:51] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:15:51] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:22] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:22] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:30] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:30] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:40] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:42] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:42] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:50] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 1 failures [02:16:50] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:01] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:24] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:24] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:24] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:25] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:25] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:25] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:26] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:27] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:30] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:31] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:41] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:44] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:51] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 1 failures [02:17:51] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:03] "Normal" puppetmaster death or something worse? [02:18:07] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:07] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:07] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:07] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:07] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:08] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:08] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:12] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:12] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:20] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:20] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:31] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:32] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:32] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:32] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:32] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:41] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:44] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 1 failures [02:18:50] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:00] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:00] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:01] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:01] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:01] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:01] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:11] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:11] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:11] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:11] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:12] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:12] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:12] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:20] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:21] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:30] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:41] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:43] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:44] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:45] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:46] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:46] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:47] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:47] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:47] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:47] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:54] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:54] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 1 failures [02:19:54] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:00] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:00] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:00] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:00] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:00] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:18] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:18] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:19] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:20:19] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:19] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:19] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:19] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:20] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:20] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:32] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:35] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:36] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:36] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:49] !log l10nupdate Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 02s) [02:20:52] !log LocalisationUpdate completed (1.25wmf9) at 2014-11-27 02:20:52+00:00 [02:20:53] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:58] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Puppet has 1 failures [02:20:58] Logged the message, Master [02:21:00] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:00] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:00] Logged the message, Master [02:21:10] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:11] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:20] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:20] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:30] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:40] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Puppet has 1 failures [02:21:50] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:00] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:00] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:00] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:01] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:10] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:30] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:41] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:52] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 1 failures [02:22:52] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Puppet has 1 failures [02:26:21] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:27:12] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:27:21] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [02:27:33] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [02:27:41] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:27:43] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [02:27:43] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:27:53] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:27:53] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [02:28:01] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [02:28:01] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [02:28:22] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:28:31] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:28:31] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:28:31] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [02:28:31] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:28:41] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [02:28:43] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:28:43] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:28:52] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:12] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [02:29:12] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:13] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [02:29:13] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:29:22] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:29:22] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [02:29:22] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:22] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:31] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:29:31] (03CR) 10Cscott: [C: 031] Next round of OCG apparmor fixes [puppet] - 10https://gerrit.wikimedia.org/r/176202 (owner: 10GWicke) [02:29:41] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:29:41] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:29:43] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:29:43] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:44] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:29:53] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [02:29:53] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:54] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:29:54] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:02] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:03] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:30:12] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:12] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:12] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:30:12] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [02:30:12] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:13] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:25] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:28] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:31] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [02:30:31] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:31] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:42] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:43] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:43] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:30:43] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [02:30:43] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:30:52] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [02:30:53] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [02:31:04] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:04] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:31:05] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:05] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:05] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:05] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:31:06] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:31:06] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:06] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:13] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:31:13] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:31:14] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:14] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [02:31:14] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:14] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:14] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:31:15] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:15] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:31:23] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:23] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:31:23] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:23] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:23] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:31:33] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [02:31:34] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:31:34] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:42] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [02:31:43] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:52] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:31:52] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:31:52] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:31:52] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:06] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:32:06] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:06] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:13] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:32:13] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:32:23] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:23] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:37] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:37] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:32:43] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:43] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:53] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:33:02] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:33:02] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:33:02] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:33:12] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:33:23] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [02:33:23] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:33:30] !log l10nupdate Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) [02:33:34] Logged the message, Master [02:33:35] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:33:35] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [02:33:36] !log LocalisationUpdate completed (1.25wmf10) at 2014-11-27 02:33:36+00:00 [02:33:39] Logged the message, Master [02:33:44] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:35:34] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:36:44] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 1 failures [02:47:14] PROBLEM - Router interfaces on mr1-esams is CRITICAL: CRITICAL: host 91.198.174.247, interfaces up: 36, down: 1, dormant: 0, excluded: 1, unused: 0BRge-0/0/0: down - Core: msw-oe12-esamsBR [02:47:35] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:59:19] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail [03:15:15] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:30:20] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:04] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:04] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:05] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:14] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:24] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:28] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:34] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 1 failures [03:32:44] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:08] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:19] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:34] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:35] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:36] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:55] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures [03:33:56] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:10] (03PS1) 10Springle: switch remaining m2 boxen to role::mariadb::misc [puppet] - 10https://gerrit.wikimedia.org/r/176216 [03:34:16] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:17] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:24] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:25] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:27] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:31] too early for puppet o'clock... [03:34:43] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:43] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 1 failures [03:35:01] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 1 failures [03:35:01] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 1 failures [03:35:38] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 1 failures [03:35:39] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 1 failures [03:35:53] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:22] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:23] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:24] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:24] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 1 failures [03:37:14] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 1 failures [03:37:41] apt timeout [03:37:47] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [03:37:49] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 1 failures [03:38:15] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 1 failures [03:38:16] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 1 failures [03:38:24] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 1 failures [03:40:39] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:41:24] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:43:16] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:43:17] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:43:25] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:43:37] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:43:56] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:44:50] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:45:10] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [03:45:18] !log puppet failures everywhere; transient apt timeout [03:45:22] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:45:22] Logged the message, Master [03:45:42] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:45:51] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [03:46:00] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:46:00] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:46:12] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:46:23] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:46:23] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:46:27] (03CR) 10Springle: [C: 032] switch remaining m2 boxen to role::mariadb::misc [puppet] - 10https://gerrit.wikimedia.org/r/176216 (owner: 10Springle) [03:46:41] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:46:51] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:47:01] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:47:01] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:47:12] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:47:31] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:47:31] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:47:40] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:48:21] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:48:24] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:49:02] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:49:50] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:50:17] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [03:50:18] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:50:20] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:51:11] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:51:42] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:52:17] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:52:17] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:52:20] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:53:27] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 1 failures [04:07:13] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:09:00] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:21:26] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:46] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:48] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:55] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:55] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:55] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:55] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet has 1 failures [04:21:59] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 1 failures [04:22:32] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 1 failures [04:22:32] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 1 failures [04:22:50] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:05] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:07] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:10] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:16] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:16] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:26] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 27 04:23:45 UTC 2014 (duration 23m 44s) [04:23:46] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:47] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:49] Logged the message, Master [04:23:57] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 1 failures [04:23:57] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:17] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:28] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:36] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:37] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:37] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:49] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 1 failures [04:24:49] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:06] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:07] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:29] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:31] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:35] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:36] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:36] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:36] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:36] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:36] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:52] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 1 failures [04:25:53] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:06] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:07] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:13] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:26] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:27] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:27] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:36] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:36] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:37] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 1 failures [04:26:58] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:22] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:22] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:22] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:26] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:32] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:38] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:42] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:46] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:57] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:58] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 1 failures [04:27:58] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 1 failures [04:28:06] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Puppet has 1 failures [04:28:46] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 1 failures [04:33:03] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:33:35] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:33:45] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:33:54] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:34:24] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:24] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:34:33] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:34:34] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:43] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:34:53] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:35:02] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:35:03] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:03] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:35:04] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:35:04] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [04:35:10] !log restarted squid3 on carbon, but glitches seem to be upstream [04:35:14] Logged the message, Master [04:35:15] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [04:35:15] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [04:35:24] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:35] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [04:35:37] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:35:41] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:44] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:35:56] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:56] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:56] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:56] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [04:35:56] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:57] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:35:57] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:35:58] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:35:58] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:04] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:05] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:14] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:15] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:15] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:16] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:34] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:40] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:42] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:42] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:42] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:44] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:45] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:45] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:36:45] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:02] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:02] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:02] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:02] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:05] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:18] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:19] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:19] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:20] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:35] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:36] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:36] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:36] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:36] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:36] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:38] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:19] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:53:44] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [06:10:08] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:33:12] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:14] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:25] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [06:33:41] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:42] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:47] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:54] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:02] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:02] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:12] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:44] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:46] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:51] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:12] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:35] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:58] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:59] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:13] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:39:25] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 339800 msg: ocg_render_job_queue 0 msg [06:47:39] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:13] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:49:32] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:50:05] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:52:45] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:14] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:33] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:35] <_joe_> we have 72 criticals, and no one cared [06:53:38] <_joe_> great [06:53:46] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:53] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:54] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:03] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:06] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:06] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:12] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:24] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:43] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:53] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:56] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:56] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:56] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:57] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:16] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:16] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:22] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:23] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:23] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:43] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:44] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:44] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:44] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:04] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:04] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:06] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:10] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:10] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:11] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:11] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:22] <_joe_> those are apt failures btw [06:56:25] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:26] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:39] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:53] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:02] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:03] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:03] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:13] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:33] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:34] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:46] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:50] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:55] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:57:55] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:08] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:17] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:17] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:25] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:05] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:05] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:22] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:24] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:24] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:37] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:57] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:19] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:30] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:46] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 1 failures [07:03:37] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:03:46] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:04:10] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:04:10] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:04:10] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:04:23] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:04:32] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:04:43] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:04:43] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:04:43] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:04:43] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:04:52] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:04] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:11] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:16] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:05:17] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:30] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:31] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:05:41] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:05:42] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:05:42] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:05:52] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:05:52] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:05] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:06:23] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:06:36] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:06:36] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:36] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:36] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:37] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:37] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:37] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:06:38] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:45] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:06:46] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:06:48] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:59] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:01] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:07:01] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:07:01] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:01] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:01] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:07:02] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:15] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:16] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:07:16] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:16] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:27] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:27] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:27] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:07:36] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:07:44] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:07:44] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:07:56] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:07:56] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:06] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:08:06] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:08:06] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:08:06] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:15] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:22] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:08:22] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:36] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:36] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:36] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:36] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:37] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:37] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:08:46] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:08:59] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:09:04] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:09:37] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:09:56] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:11:43] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:14:51] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:22:53] (03PS1) 10Giuseppe Lavagetto: ocg: fix the redis_password errors [puppet] - 10https://gerrit.wikimedia.org/r/176227 [08:28:43] the purge for thumbs seems broken [08:29:37] hm, now back o_O [08:44:01] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:28] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 58820 bytes in 0.496 second response time [08:48:43] (03PS2) 10Legoktm: Only enable Extension:Oversight on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/169611 (https://bugzilla.wikimedia.org/60373) (owner: 10Reedy) [08:48:48] (03CR) 10Legoktm: [C: 031] Only enable Extension:Oversight on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/169611 (https://bugzilla.wikimedia.org/60373) (owner: 10Reedy) [08:58:49] PROBLEM - HHVM busy threads on mw1221 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [08:59:38] PROBLEM - HHVM busy threads on mw1227 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [09:01:49] RECOVERY - HHVM busy threads on mw1221 is OK: OK: Less than 1.00% above the threshold [76.8] [09:02:18] RECOVERY - HHVM busy threads on mw1227 is OK: OK: Less than 1.00% above the threshold [76.8] [09:15:11] (03PS2) 10Giuseppe Lavagetto: labs: make mwyaml hiera backend first lookup [puppet] - 10https://gerrit.wikimedia.org/r/176204 (owner: 10BryanDavis) [09:16:36] (03CR) 10Giuseppe Lavagetto: [C: 032] "This was on the list of things I had to amend, thanks for doing it." [puppet] - 10https://gerrit.wikimedia.org/r/176204 (owner: 10BryanDavis) [09:17:34] (03CR) 10Yuvipanda: "Hmm, the fail would make sure that we don't get a recurrance of this problem, but unsure why this would work." [puppet] - 10https://gerrit.wikimedia.org/r/176227 (owner: 10Giuseppe Lavagetto) [09:19:36] (03CR) 10Giuseppe Lavagetto: "because we define (in the private hieradata) $::ocg::redis::password to be equal to $::passwords::redis::something, so making sure we incl" [puppet] - 10https://gerrit.wikimedia.org/r/176227 (owner: 10Giuseppe Lavagetto) [09:35:20] (03PS1) 10Filippo Giunchedi: codfw-prod: add ms-be2013 [software/swift-ring] - 10https://gerrit.wikimedia.org/r/176231 [09:36:31] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] codfw-prod: add ms-be2013 [software/swift-ring] - 10https://gerrit.wikimedia.org/r/176231 (owner: 10Filippo Giunchedi) [09:36:45] greetings [09:55:41] (03PS1) 10Filippo Giunchedi: "make deploy" safer by default, require DESTHOST [software/swift-ring] - 10https://gerrit.wikimedia.org/r/176233 [09:56:36] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] "make deploy" safer by default, require DESTHOST [software/swift-ring] - 10https://gerrit.wikimedia.org/r/176233 (owner: 10Filippo Giunchedi) [10:02:53] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 1 failures [10:04:26] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 1 failures [10:08:18] (03PS4) 10Giuseppe Lavagetto: reimage: add a few configs, beautify output [puppet] - 10https://gerrit.wikimedia.org/r/175965 [10:15:16] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 1 failures [10:18:14] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:26:48] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [10:30:47] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:22] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 58820 bytes in 0.132 second response time [10:36:37] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [10:59:03] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: puppet fail [10:59:41] (03PS2) 10Giuseppe Lavagetto: ocg: fix the redis_password errors [puppet] - 10https://gerrit.wikimedia.org/r/176227 [11:01:13] (03PS1) 10Giuseppe Lavagetto: Revert "ocg: Temp hack to bypass hiera for redis passwords" [puppet] - 10https://gerrit.wikimedia.org/r/176236 [11:02:32] (03PS2) 10Giuseppe Lavagetto: Revert "ocg: Temp hack to bypass hiera for redis passwords" [puppet] - 10https://gerrit.wikimedia.org/r/176236 [11:02:42] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Revert "ocg: Temp hack to bypass hiera for redis passwords" [puppet] - 10https://gerrit.wikimedia.org/r/176236 (owner: 10Giuseppe Lavagetto) [11:07:55] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Puppet has 105 failures [11:08:53] PROBLEM - swift-account-replicator on ms-be2015 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-replicator [11:09:40] <_joe_> godog: can I screw around a little with wmf-reimage? [11:10:01] <_joe_> If you're using it right now, I can abstain [11:10:18] _joe_: nope go ahead, I'm done! [11:10:23] thanks for asking tho [11:10:24] <_joe_> ok thanks [11:10:47] <_joe_> do you want to take a look at the patch btw? the next step will be rewriting it in python [11:11:03] ah yes let me do that, I noticed a couple of things too these days [11:11:33] <_joe_> https://gerrit.wikimedia.org/r/#/c/175965/ [11:12:30] +1 to python, I was imagining something like a main command with subcommands to be able to do specific steps too, e.g. "clear-keys" or "sign-keys" etc [11:12:41] <_joe_> exactly [11:12:57] <_joe_> well, my whole idea is to make it fully automated [11:13:18] <_joe_> with these changes I've made, we're nearly there [11:14:09] <_joe_> paravoid was also playing with openwsman and a python library to interact with that - it would allow us to reconfigure the BIOS automatically [11:15:03] hehe I'm expecting some pain there, but it'd be great [11:15:38] <_joe_> oh he managed to make it work I think [11:15:51] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:16:18] <_joe_> I'd happily work on that before I have to reconfigure hyperthreading on 200 servers [11:16:21] <_joe_> :P [11:17:46] heheh yeah not fun [11:19:04] (03CR) 10Filippo Giunchedi: [C: 031] "unrelated comment, the rest +1" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/175965 (owner: 10Giuseppe Lavagetto) [11:19:18] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [11:20:09] RECOVERY - swift-account-replicator on ms-be2015 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-replicator [11:22:15] (03PS1) 10Filippo Giunchedi: codfw-prod: add ms-be2015 [software/swift-ring] - 10https://gerrit.wikimedia.org/r/176239 [11:22:38] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] codfw-prod: add ms-be2015 [software/swift-ring] - 10https://gerrit.wikimedia.org/r/176239 (owner: 10Filippo Giunchedi) [11:30:40] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 1 failures [11:30:51] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:30] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:30] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:31] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:31] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:32] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:58] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:58] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:59] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 1 failures [11:31:59] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:12] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:12] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:14] puppet Y U NO WORK [11:32:21] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:23] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:23] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:29] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:30] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:30] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:38] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:49] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:49] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:49] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:50] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:50] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:50] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:05] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:09] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:10] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:10] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:10] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:20] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:20] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:20] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:20] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:32] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:32] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:32] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:43] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:43] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:43] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:51] PROBLEM - puppet last run on curium is CRITICAL: CRITICAL: Puppet has 1 failures [11:33:51] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 1 failures [11:34:02] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: Puppet has 1 failures [11:34:04] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 1 failures [11:34:10] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 1 failures [11:34:23] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 1 failures [11:34:41] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 1 failures [11:34:52] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:01] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:01] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:01] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:01] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:16] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:16] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:16] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:35] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:36] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 1 failures [11:35:51] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 1 failures [11:36:02] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 1 failures [11:36:34] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 1 failures [11:36:36] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 1 failures [11:36:51] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 1 failures [11:41:03] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [11:41:34] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [11:42:02] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [11:42:03] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [11:42:13] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:42:21] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [11:43:02] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:02] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:02] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:02] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:16] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:42] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:43:42] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:42] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:43:43] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:44:13] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:44:22] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:44:34] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [11:44:48] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [11:44:48] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:44:48] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:44:52] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [11:44:52] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:44:52] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [11:45:03] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:45:11] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [11:45:11] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [11:45:22] RECOVERY - puppet last run on curium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:45:22] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [11:45:23] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [11:45:31] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [11:45:32] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [11:45:51] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:06] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:14] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:26] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:26] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:26] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:26] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [11:46:36] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:36] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:36] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:36] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:48] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:56] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:56] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:56] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:56] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:57] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:46:57] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:06] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:06] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:06] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:06] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:07] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:16] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:19] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:26] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:29] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:38] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:47:56] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:48:06] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:48:09] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:06:43] (03Abandoned) 10Giuseppe Lavagetto: ocg: fix the redis_password errors [puppet] - 10https://gerrit.wikimedia.org/r/176227 (owner: 10Giuseppe Lavagetto) [12:12:35] PROBLEM - HHVM queue size on mw1222 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [80.0] [12:12:53] PROBLEM - HHVM busy threads on mw1222 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [12:15:20] _joe_: what is a "mainrole"? [12:15:21] RECOVERY - HHVM queue size on mw1222 is OK: OK: Less than 1.00% above the threshold [10.0] [12:15:33] RECOVERY - HHVM busy threads on mw1222 is OK: OK: Less than 1.00% above the threshold [76.8] [12:15:59] <_joe_> paravoid: just a label for grouping servers [12:16:07] <_joe_> call it whatever you prefer [12:16:22] <_joe_> 'cluster' would be better, but we abuse that already in puppet :) [12:20:17] but where do we use it? [12:20:30] git grepping says nowhere? [12:21:57] <_joe_> not right now [12:23:43] <_joe_> right now we just set it in hiera, at the node level (or in regex.yaml), so that hiera can load the correct mainrole/$mainrole.yaml file [12:24:14] <_joe_> the other solution it to set it at the node level as a variable and make hiera use that directly [12:43:32] _joe_: can't we just make our hiera backend lookup hieradata/role/foo/bar.pp? [12:43:45] based on the role class name? [12:44:04] (or without the role/ part maybe?) [12:44:37] <_joe_> how would you tell hiera to search there for data about class bar::baz ? [12:45:15] <_joe_> hiera has no info on the file or the node it's being called from apart from the scope it gets passed [12:45:40] <_joe_> which AFAIR is just all the defined variables in the node you're compiling for [12:47:25] <_joe_> for variables of role::foo we already search in $site/role/foo.yaml and common/role/foo.yaml [12:47:42] <_joe_> the issue here is that you probably want to configure classes in modules [12:48:15] Puppet sets two variables that aren’t available in regular Puppet manifests: [12:48:18] calling_module — The module in which the lookup is written. This has the same value as the Puppet $module_name variable. calling_class — The class in which the lookup is evaluated. If the lookup is written in a defined type, this is the class in which the current instance of the defined type is declared. [12:48:34] <_joe_> yeah I was reading that [12:48:42] <_joe_> let me experiment with that a little [12:49:17] so e.g. role::appserver could lookup hieradata/(role/)appserver.yaml in which we could set class parameters for class mediawiki [12:49:20] (random example) [12:49:45] I'm not sure if that makes sense outside of our role hierarchy [12:49:46] <_joe_> paravoid: yes I think I tested that and something didn't work as we (me and you would expect) [12:49:59] <_joe_> but lemme test it [12:50:16] it says it was broken between puppet 3.0 and 3.3, amazing [12:51:04] <_joe_> ok [12:52:10] <_joe_> we're on 3.4 so we should be good [12:54:40] brb [12:55:20] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [13:00:58] PROBLEM - Host text-lb.esams.wikimedia.org is DOWN: CRITICAL - Network Unreachable (91.198.174.192) [13:01:28] <_joe_> uhm [13:01:32] PROBLEM - Host cp3010 is DOWN: CRITICAL - Plugin timed out after 15 seconds [13:01:37] <_joe_> paravoid, mark__ ^^ [13:01:47] ημμ [13:01:49] RECOVERY - Host cp3010 is UP: PING OK - Packet loss = 0%, RTA = 98.05 ms [13:01:49] xmm [13:02:17] <_joe_> akosiaris: I can still see the wikis [13:02:25] mhhh eqiad<->esams blip? [13:02:27] Trouble in paradise? I'm not getting a connection to any of the sites in Europe [13:03:00] <_joe_> multichill: you are in europe and can't connect to any of the wmf sites? [13:03:01] weird, bast1001 can ping text-lb.esams [13:03:21] <_joe_> http://gdash.wikimedia.org/dashboards/reqerror/ [13:03:28] _joe_: Yup, I'm using evams [13:03:37] <_joe_> it was a one-minute glitch I'd say [13:03:38] RECOVERY - Host text-lb.esams.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 95.73 ms [13:03:42] <_joe_> multichill: still happening? [13:03:42] yeah [13:03:47] here's the recovery [13:03:50] <_joe_> I can see the sites just fine [13:05:56] Now back up [13:07:32] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 2 failures [13:07:32] <_joe_> paravoid: unfortunately, that doesn't work. The module_name variable would always be "role", while the calling class is the explicit class [13:07:34] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 1 failures [13:07:38] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [500.0] [13:08:49] csw2-esams in trouble [13:09:14] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: puppet fail [13:10:00] <_joe_> so say that we do "include nutcracker" in role::mediawiki::appserver, calling_module will be 'role', calling_class will be 'nutcracker' [13:10:16] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 2 failures [13:12:22] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:17:06] _joe_: flap between cr1-esams and cr2-knams [13:18:21] well [13:18:26] because csw2-esams is between them [13:18:27] and had issues [13:18:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [13:19:21] fpc5 ? [13:20:07] i'm not entirely sure [13:20:23] first errors: [13:20:24] Nov 27 12:59:58 csw2-esams /kernel: Buffer management parity error detected in mpfe1, value 0x90001, re-init the PFE [13:20:25] Nov 27 13:00:03 csw2-esams /kernel: peer_input_pending_internal:[4213] VKS0 for peer type 23 indx 4 reported a so_error 54 [13:21:24] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:21:24] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:21:36] back [13:23:03] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:50] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:53:46] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [13:55:06] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 1 failures [14:04:39] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [14:11:22] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [14:14:41] _joe_: the *calling* class will be nutcracker? [14:14:50] sorry just saw that [14:48:55] !log upgrading librsvg throughout the fleet [14:49:02] Logged the message, Master [14:55:59] (03PS1) 10QChris: Move gerrit's remaining ITS templates into gerrit module [puppet] - 10https://gerrit.wikimedia.org/r/176264 [14:56:01] (03PS1) 10QChris: Remove hooks-bugzilla configuration [puppet] - 10https://gerrit.wikimedia.org/r/176265 [14:56:03] (03PS1) 10QChris: Drop 'Phabricator' suffix from gerrit's ITS actions [puppet] - 10https://gerrit.wikimedia.org/r/176266 [14:56:05] (03PS1) 10QChris: Switch Gerrit's 'Report Bug' url to Phabricator [puppet] - 10https://gerrit.wikimedia.org/r/176267 [15:06:34] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 1 failures [15:06:54] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 1 failures [15:06:58] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:05] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:18] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:19] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:20] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:20] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:23] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:23] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:30] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:32] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:43] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:45] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:46] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:46] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:53] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:54] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:54] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:54] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:54] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:55] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:07] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:09] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:09] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:14] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:23] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:24] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:24] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:24] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:33] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:33] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:45] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 1 failures [15:08:56] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:01] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:01] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:01] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:01] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:01] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:02] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:02] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:10] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:12] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:37] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:37] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:41] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:51] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 1 failures [15:09:52] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:00] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:14] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:14] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:15] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:22] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:31] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:31] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:31] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:40] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 1 failures [15:10:51] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 1 failures [15:11:02] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 1 failures [15:11:02] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 1 failures [15:11:23] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: Puppet has 1 failures [15:11:31] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 1 failures [15:11:43] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet has 1 failures [15:12:12] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 1 failures [15:12:42] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 1 failures [15:12:50] PROBLEM - Apache HTTP on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:51] PROBLEM - HHVM rendering on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:19] PROBLEM - HHVM queue size on mw1224 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [80.0] [15:15:45] PROBLEM - HHVM busy threads on mw1224 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [115.2] [15:17:15] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [15:17:25] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:18:25] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:18:32] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:18:33] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:18:46] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [15:18:46] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:18:47] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:18:47] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:18:48] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [15:18:48] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [15:18:55] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [15:19:04] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:19:04] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:19:16] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [15:19:16] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:19:16] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:19:25] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:19:25] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:19:25] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [15:19:25] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:19:26] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [15:19:26] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:19:34] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:34] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:34] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [15:19:34] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:19:34] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:53] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:54] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:54] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:54] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:55] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [15:19:55] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:19:56] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [15:19:56] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [15:20:05] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:20:06] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:20:06] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:20:16] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:16] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:27] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:20:32] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:33] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:33] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:45] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:45] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:46] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:20:46] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:21:06] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:06] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:18] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:18] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:18] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:26] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:26] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:35] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [15:21:37] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:38] gj puppet [15:21:45] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:46] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:21:55] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:22:01] <_joe_> this is a consequence of apt-get update failing [15:22:05] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:22:05] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:22:05] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:22:17] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:28:42] (03PS1) 10Hoo man: Set "displayStatementsOnProperties" for wikidata/testwikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176271 [15:28:49] (03PS1) 10Glaisher: Add 'move-subpages' right to "closer" and "filemover" groups at ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176272 [15:29:43] (03PS1) 10Glaisher: Add 'move-subpages' right to "closer" and "filemover" user groups at ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176273 [15:30:11] wtf [15:30:22] (03Abandoned) 10Glaisher: Add 'move-subpages' right to "closer" and "filemover" user groups at ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176273 (owner: 10Glaisher) [15:34:06] (03CR) 10Hoo man: [C: 032] Set "displayStatementsOnProperties" for wikidata/testwikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176271 (owner: 10Hoo man) [15:34:16] (03Merged) 10jenkins-bot: Set "displayStatementsOnProperties" for wikidata/testwikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176271 (owner: 10Hoo man) [15:34:49] !log hoo Synchronized wmf-config/Wikibase.php: Set "displayStatementsOnProperties" for wikidata/testwikidata (duration: 00m 06s) [15:34:52] Logged the message, Master [15:48:02] !log hoo Synchronized php-1.25wmf10/extensions/Wikidata/: Fixing a data model bug + enable Statements on Properties for testwikidata (duration: 00m 12s) [15:48:06] Logged the message, Master [16:01:50] (03PS1) 10Glaisher: Restore default configuration for ruwikisource bureaucrats [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176287 [16:22:37] (03PS1) 10Giuseppe Lavagetto: graphite/txstatsd: re-introduce require_package [puppet] - 10https://gerrit.wikimedia.org/r/176289 [16:23:21] <_joe_> godog: I think I found a solution for using require_package and "require" [16:24:49] _joe_: nice, interestingly enough the real solution came to me last night, the txstatsd package really is missing the dependency [16:25:32] <_joe_> godog: lol, true [16:25:57] <_joe_> but I just wanted to do this to beat the bug :) [16:26:21] yeah we might need it to use it regardless of this [16:29:35] (03PS2) 10Giuseppe Lavagetto: graphite/txstatsd: re-introduce require_package [puppet] - 10https://gerrit.wikimedia.org/r/176289 [16:29:48] <_joe_> we can give it a shot maybe [16:32:01] yep why not [16:33:25] (03CR) 10Giuseppe Lavagetto: [C: 032] graphite/txstatsd: re-introduce require_package [puppet] - 10https://gerrit.wikimedia.org/r/176289 (owner: 10Giuseppe Lavagetto) [16:33:29] <_joe_> let's try [16:42:33] (03PS2) 10Aklapper: Add 'move-subpages' right to "closer" and "filemover" groups at ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176272 (owner: 10Glaisher) [16:48:47] !log upload missing txstatsd 1.0.0-1 to carbon [16:48:51] Logged the message, Master [16:49:10] !log upload missing txstatsd 1.0.0-1 _source package_ to carbon [16:49:13] Logged the message, Master [16:54:17] (03PS2) 10Revi: Restore default configuration for ruwikisource bureaucrats [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176287 (owner: 10Glaisher) [17:00:07] (03PS3) 10Revi: Restore default configuration for ruwikisource bureaucrats [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176287 (owner: 10Glaisher) [17:01:08] (03CR) 10Revi: "Patchset 3 should send notification to Phab. (Patchset 2 was mistake)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/176287 (owner: 10Glaisher) [17:09:05] !log upload txstatsd 0.7.0~bzr30-0ubuntu0+14 to precise-wikimedia on carbon [17:09:09] Logged the message, Master [17:17:59] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 1 failures [17:24:12] !log removed /var/lib/carbon/whisper/archived/jenkins from tungsten [17:24:14] Logged the message, Master [17:28:47] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:37:31] PROBLEM - HHVM busy threads on mw1233 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [17:38:17] PROBLEM - HHVM busy threads on mw1232 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [17:40:22] RECOVERY - HHVM busy threads on mw1233 is OK: OK: Less than 1.00% above the threshold [76.8] [17:41:02] RECOVERY - HHVM busy threads on mw1232 is OK: OK: Less than 1.00% above the threshold [76.8] [17:48:42] PROBLEM - HHVM busy threads on mw1233 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [17:49:29] <_joe_> that looks like it's begging for a restart [17:49:39] PROBLEM - HHVM busy threads on mw1231 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [17:51:22] RECOVERY - HHVM busy threads on mw1233 is OK: OK: Less than 1.00% above the threshold [76.8] [17:52:24] RECOVERY - HHVM busy threads on mw1231 is OK: OK: Less than 1.00% above the threshold [76.8] [17:54:23] <_joe_> we do have one stuck hhvm server in the api pool, almost all threads are stuck on https://phabricator.wikimedia.org/P112 [17:55:07] RECOVERY - Apache HTTP on mw1224 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.047 second response time [17:55:08] RECOVERY - HHVM rendering on mw1224 is OK: HTTP OK: HTTP/1.1 200 OK - 68454 bytes in 0.397 second response time [17:55:43] <_joe_> !log restarted hhvm on mw1224, the alarm may have been lost in the puppet failure shower earlier [17:55:51] Logged the message, Master [18:05:19] RECOVERY - HHVM queue size on mw1224 is OK: OK: Less than 1.00% above the threshold [10.0] [18:05:33] RECOVERY - HHVM busy threads on mw1224 is OK: OK: Less than 1.00% above the threshold [76.8] [19:18:29] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: Puppet has 1 failures [19:18:29] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 1 failures [19:18:40] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 1 failures [19:18:40] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 1 failures [19:18:47] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:00] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:12] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:38] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:39] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:40] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:47] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:58] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 1 failures [19:19:59] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 1 failures [19:20:08] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 1 failures [19:20:19] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Puppet has 1 failures [19:20:20] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 1 failures [19:20:27] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: Puppet has 1 failures [19:20:28] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 1 failures [19:20:38] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 1 failures [19:21:02] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Puppet has 1 failures [19:21:02] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: Puppet has 1 failures [19:21:08] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [19:21:29] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 1 failures [19:22:28] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 1 failures [19:29:37] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:29:45] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:29:48] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:29:55] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:30:05] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:16] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:30:21] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:30:34] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:30:36] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:45] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [19:30:46] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:56] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:30:56] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:31:23] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:31:24] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:31:25] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:31:28] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [19:31:28] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:31:45] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:32:03] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:32:08] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:32:09] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:32:29] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:30] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:35:17] (03PS1) 10Legoktm: extdist: clone composer into /srv/composer [puppet] - 10https://gerrit.wikimedia.org/r/176294 [19:37:33] legoktm: 1. use $base_dir/composer, 2. ensure the dir exists already (line 51), 3. dependency on folder existing (line 61) [19:38:04] ok [19:38:12] * legoktm will be afk for a bit [19:44:56] SCANDAL [19:48:22] * YuviPanda gives Nemo_bis a token [19:48:39] * Nemo_bis inserts coin [20:00:30] (03PS2) 10Legoktm: extdist: clone composer into /srv/composer [puppet] - 10https://gerrit.wikimedia.org/r/176294 [20:00:34] YuviPanda: ^ all done [20:02:42] (03CR) 10Yuvipanda: [C: 032] extdist: clone composer into /srv/composer [puppet] - 10https://gerrit.wikimedia.org/r/176294 (owner: 10Legoktm) [20:02:48] :D [20:02:55] legoktm: :) [20:03:07] * legoktm starts puppet run [20:05:07] YuviPanda: uhhhhh [20:05:13] errored? [20:05:15] YuviPanda: this host doesn't have php installed >.> [20:05:24] hehe :) [20:05:25] install it? [20:05:29] yeah, one sec [20:06:09] legoktm: I mean, via puppet :) [20:06:16] YuviPanda: do I just do require => Package['php-cli'] ? [20:06:41] legoktm: yeah, but before that do a package { 'php-cli': ensure => present } [20:08:20] ok [20:09:24] YuviPanda: the cassandra module already uses puppet [20:09:30] eh, hiera [20:09:41] (03PS1) 10Legoktm: extdist: composer depends on php5-cli [puppet] - 10https://gerrit.wikimedia.org/r/176295 [20:09:46] YuviPanda: ^ [20:10:02] gwicke: but also uses global variables :) [20:10:30] (03CR) 10Yuvipanda: [C: 032] extdist: composer depends on php5-cli [puppet] - 10https://gerrit.wikimedia.org/r/176295 (owner: 10Legoktm) [20:10:31] afaik it's all hiera [20:10:48] gwicke: not really. default.pp uses pick and global variables. [20:10:55] that's the way I have used it so far at least [20:11:04] the global vars are all injected by hiera [20:11:13] yeah, they shouldn't be global variables. [20:11:18] there's an extra layer of indirection there [20:11:26] should just be params to the cassandra main class [20:12:17] I don't see how that makes a difference to how hiera is used [20:12:37] is this merely a simplification by getting rid of the separate defaults.pp? [20:13:27] yes, it's just a code smell [20:13:37] default.pp shouldn't exist [20:14:45] I see [20:15:25] might make sense to clarify that in the https://phabricator.wikimedia.org/T76149 summary [20:15:45] am happy to do that if you don't mind [20:16:21] gwicke: done [20:16:23] gwicke: sure! [20:16:45] looks good, thanks! [20:17:14] added Andrew in the CC as well, as he wrote the cassandra module [20:20:11] legoktm@extdist2:/srv/composer$ vendor/bin/composer --version [20:20:11] Composer version 1.0.0-alpha8 @release_date@ [20:20:13] YuviPanda: :D ty! [20:20:52] gwicke: cool :) [21:04:11] (03PS1) 10Yuvipanda: tools: Update update-scripts.sh to update updated Packages properly [puppet] - 10https://gerrit.wikimedia.org/r/176302 [21:04:13] Coren: ^ +1? [21:05:06] (03CR) 10coren: [C: 031] "Yeah, I never found a rationale for this change of behaviour from apt-get, but this neatly works around it." [puppet] - 10https://gerrit.wikimedia.org/r/176302 (owner: 10Yuvipanda) [21:05:26] today is very quiet [21:05:26] (03CR) 10Yuvipanda: [C: 032] tools: Update update-scripts.sh to update updated Packages properly [puppet] - 10https://gerrit.wikimedia.org/r/176302 (owner: 10Yuvipanda) [21:05:29] I wonder what's up [21:55:04] PROBLEM - HHVM busy threads on mw1228 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [21:55:05] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: puppet fail [21:55:41] PROBLEM - HHVM busy threads on mw1229 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [21:58:24] RECOVERY - HHVM busy threads on mw1229 is OK: OK: Less than 1.00% above the threshold [76.8] [22:00:43] RECOVERY - HHVM busy threads on mw1228 is OK: OK: Less than 1.00% above the threshold [76.8] [22:11:39] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:30:42] PROBLEM - HHVM busy threads on mw1232 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [22:32:54] PROBLEM - HHVM busy threads on mw1233 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [22:35:35] RECOVERY - HHVM busy threads on mw1233 is OK: OK: Less than 1.00% above the threshold [76.8] [22:36:24] RECOVERY - HHVM busy threads on mw1232 is OK: OK: Less than 1.00% above the threshold [76.8] [22:41:14] is there an ETA for having phab 100% merged from bz? [22:44:43] merged from? [22:56:52] Betacommand, what's missing? [22:57:23] PROBLEM - HHVM busy threads on mw1223 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [22:57:34] Betacommand: https://www.mediawiki.org/wiki/Phabricator/versus_Bugzilla#Timeline ? [23:00:24] RECOVERY - HHVM busy threads on mw1223 is OK: OK: Less than 1.00% above the threshold [76.8] [23:19:14] andre__afk: None of the bugs Ive reported/cc on dont have be subscribed [23:19:24] Betacommand, do you have an example? [23:19:34] when did you claim your BZ account in Phab? [23:25:01] (03PS1) 10Tim Starling: Move idiosyncratic gdbinit to /home/ori [puppet] - 10https://gerrit.wikimedia.org/r/176307 [23:30:19] andre__afk: sorry I DCed [23:30:22] andre__afk: https://phabricator.wikimedia.org/T49505 is one example [23:30:31] Betacommand: when did you claim/add your BZ account in your Phab settings? [23:30:57] andre__afk: I registered the BZ email stuff before the migration [23:31:55] hmm, and I guess you verified that email [23:32:09] yep [23:32:15] Oct 24 to be exact [23:32:56] Betacommand, hopefully chasemp can take a look at this when he's back around. could you file a ticket against the Phabricator project about your case, with one example of a imported ticket that should have been already yours now? [23:33:13] sorry for the hassle :-/ [23:33:32] andre__afk: Dont worry, I seem to find all the bugs [23:33:38] Just ask Coren :P [23:33:42] still not cool :P [23:33:50] doesn't scale :D [23:34:09] andre__afk: I find the most bizarre bugs possible [23:34:16] T74931 should have been already yours, yeah [23:34:24] I like that! :D [23:34:39] andre__afk: I dont [23:35:01] andre__afk: gets kinda old after awhile [23:35:32] andre__afk: what should I use as a title? [23:37:44] err... "My Bugzilla contributions did not get claimed in Phab" or such [23:39:52] andre__afk: thanks, been a long day and my brain is only kinda working [23:40:04] heh, I can sign that for the last week too :D [23:40:27] Couldnt think of a short phrase to describe the issue [23:41:11] andre__afk: just finished a Windows and ducktape webserver migration to Ubuntu