[00:00:05] RoanKattouw ostriches: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151113T0000). [00:00:06] MatmaRex jhobs RoanKattouw: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [00:00:11] hi [00:00:17] i'm here [00:00:54] 6operations, 10Analytics, 10CirrusSearch, 6Discovery: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old. - https://phabricator.wikimedia.org/T118527#1802620 (10EBernhardson) 3NEW [00:01:00] RoanKattouw: I think this might be right: https://gerrit.wikimedia.org/r/#/c/252873/ [00:03:20] (03PS2) 10Yuvipanda: admin: Add new key for self (ori) [puppet] - 10https://gerrit.wikimedia.org/r/252866 (owner: 10Ori.livneh) [00:03:39] (03PS3) 10Yuvipanda: admin: Add new key for self (ori) [puppet] - 10https://gerrit.wikimedia.org/r/252866 (owner: 10Ori.livneh) [00:03:44] AndyRussG: Just checking, this is like a month's worth of code, right? [00:04:17] RoanKattouw: yeah, is that allowed? [00:04:44] Sure [00:04:49] Just verifying [00:08:46] 6operations, 6Reading-Admin: Improve UX Strategic Test - https://phabricator.wikimedia.org/T117826#1802660 (10dr0ptp4kt) [00:08:52] (03PS1) 10Aaron Schulz: Disable ChronologyProtector for rpc/RunJobs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252878 [00:09:44] RoanKattouw: cool! [00:10:05] K so ready to test whenever :) [00:10:13] Thanks again! [00:11:41] (03CR) 10Catrope: [C: 032] Turn off language detection user test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252867 (https://phabricator.wikimedia.org/T118197) (owner: 10EBernhardson) [00:12:03] (03Merged) 10jenkins-bot: Turn off language detection user test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252867 (https://phabricator.wikimedia.org/T118197) (owner: 10EBernhardson) [00:12:06] (03CR) 10Catrope: [C: 032] Revert "Hardcode UploadWizard max upload size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/248360 (https://phabricator.wikimedia.org/T98933) (owner: 10Bartosz Dziewoński) [00:12:54] (03Merged) 10jenkins-bot: Revert "Hardcode UploadWizard max upload size" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/248360 (https://phabricator.wikimedia.org/T98933) (owner: 10Bartosz Dziewoński) [00:14:19] !log catrope@tin Synchronized wmf-config/InitialiseSettings.php: Turn off language detection user test for CirrusSearch (duration: 00m 30s) [00:14:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:14:29] Whoa I just got a giant list of errors [00:14:34] 00:13:59 ['/srv/deployment/scap/scap/bin/sync-master', 'tin.eqiad.wmnet'] on mira.codfw.wmnet returned [70]: 00:13:49 Copying to mira.codfw.wmnet from tin.eqiad.wmnet [00:14:36] 00:13:49 Started rsync master [00:14:37] rsync: failed to set times on "/srv/mediawiki-staging/.git/modules/portals/objects/06": Operation not permitted (1) [00:14:39] rsync: failed to set times on "/srv/mediawiki-staging/.git/modules/portals/objects/16": Operation not permitted (1) [00:14:40] etc etc [00:14:44] Also cannot delete non-empty directory: php-1.26wmf13/cache/l10n and a bunch of similar ones [00:14:44] RoanKattouw: expected [00:14:49] it's ok [00:14:57] I think that's all harmless, but ideally sync-file wouldn't barf a screenful of errors in my face [00:15:14] ebernhardson: Your CirrusSearch thing is deployed, please verify [00:15:40] RoanKattouw: we are trying to figure out a good fix. The new branch messed up our fragile cross master rsync [00:16:04] !log catrope@tin Synchronized wmf-config/CommonSettings.php: Remove hardcoded max upload size for UploadWizard (duration: 00m 30s) [00:16:10] MatmaRex: ---^^ [00:16:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:17:43] RoanKattouw: thanks, all seems to be in order [00:18:18] RoanKattouw: thanks, looks good [00:18:38] (03CR) 10Tim Starling: [C: 031] "Merge when you are ready." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252878 (owner: 10Aaron Schulz) [00:23:28] !log catrope@tin Synchronized php-1.27.0-wmf.6/extensions/CentralNotice: SWAT (duration: 00m 31s) [00:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:24:00] !log catrope@tin Synchronized php-1.27.0-wmf.6/extensions/Flow: SWAT (duration: 00m 32s) [00:24:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:27:41] AndyRussG: Your CN update went out [00:27:47] About 4 mins ago, I forgot to notify earlier [00:27:59] !log catrope@tin Synchronized php-1.27.0-wmf.6/extensions/QuickSurveys: SWAT (duration: 00m 30s) [00:28:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:29:03] RoanKattouw: K checking! [00:29:41] So, I'm wondering if there is a holiday for celebrating operations people. [00:29:48] Anyone know of such a thing? [00:30:01] Because we should have one. [00:30:38] halfak: yes, there's 'sysadmin day' [00:31:03] why do you want to celebrate us halfak? :) [00:31:31] Because you guys often need to drop your work to go fix something that wasn't your fault and the rest of us benefit from that. [00:31:42] * halfak works with Yuvi a lot and gets to see him suffer. [00:31:52] 6operations, 10fundraising-tech-ops: make sure netapp fundraising share gets wiped - https://phabricator.wikimedia.org/T118535#1802733 (10Jgreen) 3NEW a:3Jgreen [00:32:16] There is both Tim Starling Day and Brion Vibbler day [00:32:29] 6operations, 10fundraising-tech-ops: make sure netapp fundraising share gets wiped - https://phabricator.wikimedia.org/T118535#1802741 (10Jgreen) [00:34:03] 6operations, 10Analytics, 10Analytics-Cluster, 10Fundraising Tech Backlog, and 3 others: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1802755 (10Jgreen) 5Open>3Resolved (07:14:09 PM) awight: Jeff_Green: so the new data is good, and is correctly multipled... [00:36:39] https://en.wikipedia.org/wiki/Wikipedia:Tim_Starling_Day [00:36:48] Just missed it (Oct. 31st) [00:37:24] https://en.wikipedia.org/wiki/Wikipedia:Brion_Vibber_Day is June 1st. [00:37:34] PROBLEM - puppet last run on mw2071 is CRITICAL: CRITICAL: puppet fail [00:37:53] (03PS1) 10EBernhardson: Sample CirrusSearchRequestSet at 1:10000 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252879 [00:38:08] don't forget https://en.wikipedia.org/wiki/Wikipedia:Magnus_Manske_Day. The day when we curse and rejoice for our PHP heritage [00:39:28] thanks whoever deployed my patch, it appears to be working! [00:42:28] bd808: but it was perl before that! [00:47:56] RoanKattouw: I guess it takes a while for all the i18n messages to propagate? I'm getting some angle brackets on the CentralNotice admin UI [00:48:10] Oh crap [00:48:15] I completely forgot :( [00:48:16] * RoanKattouw scaps [00:48:16] he just did a sync-dir [00:48:25] heh [00:48:30] !log catrope@tin Started scap: I totally forgot that the SWAT contained i18n changes [00:48:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:48:37] RoanKattouw: ahhh K well indeed that explains it! [00:48:41] thanks :) [00:49:31] It's cool actually, this way I can already test the banner display JS on prod, which is the main thing to test... [01:15:17] !log catrope@tin Finished scap: I totally forgot that the SWAT contained i18n changes (duration: 26m 46s) [01:15:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:15:45] AndyRussG, ^ [01:18:41] RoanKattouw: :-) [01:20:32] Could any ops bounce hhvm on mw1045? It's looks to have some thread stuck in an infinite loop during the post-send portion of request handling (best guess) which is then dropping log lines into fluorine fatal.log like: Nov 13 00:12:03 mw1045: message repeated 95341800 times: [ #012Notice: fwrite(): send of 61 bytes failed with errno=32 Broken pipe in /srv/mediawiki/php-1.27.0-wmf.5/vendor/nmred/kafka-php/src/Kafka/Socket [01:20:38] .php on line 330] [01:21:00] i don't think its hurting anything in particular, and the server is still serving requests as normal. It's just log spam and perhaps chewing up extra cpu. [01:21:05] Krenair: coolio... K those <> messages should be gone once RL cache becomse all shiney new [01:21:52] generally guessing based on `hhvmadm status.json` and the thread with id Nov 13 00:12:03 mw1045: message repeated 95341800 times: [ #012Notice: fwrite(): send of 61 bytes failed with errno=32 Broken pipe in /srv/mediawiki/php-1.27.0-wmf.5/vendor/nmred/kafka-php/src/Kafka/Socket [01:22:04] * ebernhardson hates copy/patse [01:22:46] generally guessing based on `hhvmadm status.json` and the thread with id 140341382477568 being in the "psp" mode for the last 20 minutes (since i found out about the logs) [01:27:23] !log bounced mw1045 HHVM for ebernhardson [01:27:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:33:44] (03PS1) 10Faidon Liambotis: Revoke John F. Lewis' access [puppet] - 10https://gerrit.wikimedia.org/r/252883 [01:37:11] (03CR) 10Faidon Liambotis: [C: 032] Revoke John F. Lewis' access [puppet] - 10https://gerrit.wikimedia.org/r/252883 (owner: 10Faidon Liambotis) [01:39:43] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet last ran 11 hours ago [01:40:53] (03PS1) 10Faidon Liambotis: Remove John F. Lewis from shinken [puppet] - 10https://gerrit.wikimedia.org/r/252884 [01:40:55] (03PS1) 10Faidon Liambotis: Kill POSIX group and contact for mailman-admins [puppet] - 10https://gerrit.wikimedia.org/r/252885 [01:42:29] (03CR) 10Faidon Liambotis: [C: 032] Remove John F. Lewis from shinken [puppet] - 10https://gerrit.wikimedia.org/r/252884 (owner: 10Faidon Liambotis) [01:42:33] (03CR) 10Faidon Liambotis: [C: 032] Kill POSIX group and contact for mailman-admins [puppet] - 10https://gerrit.wikimedia.org/r/252885 (owner: 10Faidon Liambotis) [01:43:23] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:45:33] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet last ran 11 hours ago [01:47:23] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [01:51:44] (03PS3) 10Thcipriani: Move scap-specific items out of mediawiki class [puppet] - 10https://gerrit.wikimedia.org/r/252362 (https://phabricator.wikimedia.org/T116606) [01:51:46] (03PS1) 10Thcipriani: RESTBase configuration for scap3 deployment [puppet] - 10https://gerrit.wikimedia.org/r/252887 [01:52:00] RoanKattouw: Hi! I'm still getting angle brackets on the Meta CentralNotice UI: https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&subaction=noticeDetail¬ice=SciPhotoGreece2015 [01:52:37] These are messages that added on the client, provided w/ corresponding RL modules [01:52:39] Interesting, https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-impression-diet-identifier does work [01:52:53] Does https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&subaction=noticeDetail¬ice=SciPhotoGreece2015 inject that message using JS? [01:52:59] RoanKattouw: Have RL cache times changed? [01:53:00] Yes [01:53:20] RL cache doesn't use times any more so this is a bit trickier now [01:53:31] Krinkle: You know how this stuff works, can you help? [01:54:06] I SWATted a change with sync-dir not realizing it had i18n in it, then ran scap. https://meta.wikimedia.org/wiki/MediaWiki:Centralnotice-impression-diet-identifier shows the correct message but https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice&subaction=noticeDetail¬ice=SciPhotoGreece2015 uses JS to inject it and that still displays as if the message doesn't exist [01:54:14] RoanKattouw: What? [01:55:00] Ah hmm I do recall hearing about some changes....... [01:55:01] mw.messages.get('centralnotice-impression-diet-identifier') [01:55:01] "" [01:55:04] It comes from JS indeed [01:55:06] Yeah [01:55:07] it is sent from RL [01:55:16] Probably the known bug? [01:55:26] * AndyRussG plays scary music [01:55:36] RL does use times for messages [01:55:40] Oh right [01:55:42] Module name is ext.centralNotice.adminUi.campaignManager [01:55:43] it's one of the keys in the definition hash of File modules. [01:55:53] msgBlobMTime [01:56:02] I'm working on that as it happens (see -dev) [01:56:06] "centralnotice-impression-diet-identifier":"\u003Ccentralnotice-impression-diet-identifier\u003E" [01:56:10] even on an uncached view [01:56:13] Checking logstash to see if it was reported [01:56:17] So probably something bad stuck in MessageBlobStore [01:56:19] We have logging for that bug now [01:56:31] To be fair, when I initially deployed this code, the message didn't exist [01:56:36] AKA I screwed up [01:56:37] https://phabricator.wikimedia.org/T93800 [01:56:46] But now the message's "value" has changed (really, its existence status has changed) [01:57:45] Krenair: RoanKattouw: The new messages that come from PHP are indeed OK [01:58:06] RoanKattouw: https://logstash.wikimedia.org/#dashboard/temp/AVD-kDraptxhN1XaYc_C [01:58:22] how long ago? [01:59:11] I've purged it now using on-wiki purge for that mediawiki pae [01:59:11] ge [02:00:21] we'll see in a few minutes if that works [02:00:39] OK, adding &x=1 to my load.php URL shows the correct message contents now [02:00:42] it's a race condition so hteres no fix other than to try again now that the message does exist. Either by invalidating the module's sripts or the message [02:00:48] So let's see if the startup module version turns over correctly [02:01:02] I didn't know purging the MediaWiki: page worked, that's a nice way to do it [02:01:17] Although AIUI that'll only fix that one message due to the surgical replacement code in MessageCache, right? [02:01:19] RoanKattouw: Well, bypassing varnish for that key will always work [02:01:30] that worked from a second after it happened [02:01:38] Well it didn't work before [02:01:43] Hm.. [02:01:43] Not until you purged the MW page [02:01:52] Interesting [02:01:57] Like I said, I think the blob store actually had the wrong content [02:02:01] Right [02:02:04] anyways, we'll find out soon enough [02:02:13] That would be differnet from the bug as reported there about ve and citoid previously [02:02:23] Yeah, in your case it was in blob store because deployment in the wrong order [02:02:25] And diet-idenitifier-help is still broken [02:02:26] so it required a purge to fix [02:02:26] I'll purge that one too [02:02:28] Yup [02:02:53] I accidentally created the page. Fixed now [02:03:20] Okay, regular -identifier shows up now in the normal page [02:03:24] Awesome [02:03:25] without query params [02:03:33] i.e. the startup module picked it up [02:03:37] I only just purged identifier-help so that one might take 5 more minutes [02:03:51] Per https://logstash.wikimedia.org/#/dashboard/temp/AVD-kDraptxhN1XaYc_C it seems like a lot more are missing though [02:03:56] So be sure to verify those as well [02:04:13] 9 in total, 7 without these two [02:05:09] Krenair: RoanKattouw: for CN, you can see other angle brackets by clicking on the elegantly titled "Large banner limit and switch" checkbox [02:05:28] Yeah I'm going through those 7 that Krinkle mentioned [02:06:16] * RoanKattouw enjoys GET ?action=purge while it lasts [02:06:45] Heh fun [02:07:37] "ResourceLoaderModule::saveFileDependencies: failed to update DB: exception 'DBReadOnlyError' with message 'Database is read-only: The database has been automatically locked while the slave database servers catch up to the master.' in /srv/mediawiki/php-1. " sounds a bit odd. [02:08:05] "ResourceLoaderModule::saveFileDependencies: failed to update DB: exception 'DBQueryError' with message 'A database error has occurred. Did you forget to run maintenance/update.php after upgrading? See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_ " more so. [02:08:32] OK I've purged them all now [02:08:42] AndyRussG: In ~5 minutes you should see all of those messages [02:08:49] RoanKattouw: fantastic!!! [02:09:31] Thanks again :) [02:17:21] RoanKattouw: https://logstash.wikimedia.org/#/dashboard/temp/AVD-kDraptxhN1XaYc_C [02:22:28] James_F: Those are SNAFU unfortunately. Been there for years. It's what happens when multiple requests try to make the same update. Eventually consistent, no harm. [02:22:39] But the current refactoring will naturally avoid that [02:22:43] Krinkle: Cool. [02:23:00] Krinkle: I'm curious why Commons is so prominent. Just more time with the code live there? [02:23:39] legoktm: RoanKattouw: btw, just checking, exceptions had a spike earlier from Echo maintenance script (Catchable fatal about arg passed to EchoDiscussionParser::getTextSnippet) [02:23:48] resolved/known? [02:23:54] Which maintenance script? [02:23:59] not known, which script? [02:24:03] processEchoEmailBatch [02:24:08] Started 1.5h ago [02:24:10] and then stoped [02:24:20] https://logstash.wikimedia.org/#dashboard/temp/AVD-qF4GptxhN1XaZJGX [02:24:21] Not known to me either [02:24:42] Catchable fatal error: Argument 2 passed to EchoDiscussionParser::getTextSnippet() must be an instance of Language, StubUserLang given [02:24:52] Some sort of strange unstubbing bug? [02:24:57] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 6 below the confidence bounds [02:25:03] Oh, type hints probably don't play well with stubbing [02:25:09] RoanKattouw: Krinkle: sorry 4 the bother, still angle brackets... 8p [02:25:17] AndyRussG: Which messages? [02:25:28] * legoktm files a bug [02:25:47] 'centralnotice-large-banner-limit-days', [02:25:48] 'centralnotice-large-banner-limit-days-help', [02:25:48] 'centralnotice-large-banner-limit-randomize', [02:25:48] 'centralnotice-large-banner-limit-randomize-help', [02:25:48] 'centralnotice-large-banner-limit-identifier', [02:25:50] 'centralnotice-large-banner-limit-identifier-help', [02:26:04] Hrmph those are the ones I just purged [02:26:11] * Krinkle steps out briefly for dinner [02:26:27] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 1 failures [02:26:28] (Again, click on "Large banner limit and switch" to see them [02:26:29] ) [02:26:30] Hmmm [02:26:33] !log l10nupdate@tin Synchronized php-1.27.0-wmf.6/cache/l10n: l10nupdate for 1.27.0-wmf.6 (duration: 05m 21s) [02:26:40] I can't, I don't have the rights, checking from the console instead [02:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:26:48] https://phabricator.wikimedia.org/T118542?workflow=create [02:26:51] Ah sorry [02:27:41] Hrmph [02:27:59] Krinkle: Any objections to me going in and killing the msg_resource row for that module to force a full recomputation? [02:28:19] Oh he just stepped out [02:28:22] I'll just do it then [02:28:26] :) [02:28:37] It's actually not urgent, this a new feature no one's using yet [02:28:58] Just thought I'd wait 'till it's all angleless before announcing to FR... [02:30:27] AndyRussG: Give it another ~5 mins and it should all be fixed then [02:30:48] RoanKattouw: fantastic! ;D [02:31:16] expires:Fri, 13 Nov 2015 02:32:38 GMT [02:31:20] After that timestamp, it should start working [02:31:31] i.e. a minute and change from now [02:34:53] RoanKattouw: yep all good now!!!! [02:34:58] thx again! [02:35:03] Awesome [02:43:37] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 9 below the confidence bounds [02:52:27] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:58:27] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 9 below the confidence bounds [03:04:08] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 7 below the confidence bounds [03:09:38] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 7 below the confidence bounds [03:13:18] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 9 below the confidence bounds [03:24:19] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 6 below the confidence bounds [03:35:38] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [03:59:28] PROBLEM - Outgoing network saturation on labstore1001 is CRITICAL: CRITICAL: 11.54% of data above the critical threshold [100000000.0] [04:23:37] RECOVERY - Outgoing network saturation on labstore1001 is OK: OK: Less than 10.00% above the threshold [75000000.0] [04:39:30] (03PS2) 10Thcipriani: RESTBase configuration for scap3 deployment [puppet] - 10https://gerrit.wikimedia.org/r/252887 [05:42:48] PROBLEM - Incoming network saturation on labstore1003 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [100000000.0] [05:55:38] PROBLEM - Incoming network saturation on labstore1003 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [100000000.0] [06:03:17] RECOVERY - Incoming network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [75000000.0] [06:05:47] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:07:18] PROBLEM - Router interfaces on cr2-codfw is CRITICAL: CRITICAL: host 208.80.153.193, interfaces up: 117, down: 1, dormant: 0, excluded: 1, unused: 0BRxe-5/2/1: down - Core: cr1-eqord:xe-0/0/0 (Telia, IC-314534, 24ms) {#10694} [10Gbps DWDM]BR [06:07:57] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 6 below the confidence bounds [06:15:17] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [06:15:48] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:17] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:17] PROBLEM - puppet last run on labtestnet2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:18] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:38] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:38] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:39] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:39] PROBLEM - puppet last run on db2054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:47] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:48] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:48] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:57] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:58] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:58] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:16:59] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:07] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:08] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:09] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:17] PROBLEM - puppet last run on labvirt1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:18] PROBLEM - puppet last run on labcontrol1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:18] PROBLEM - puppet last run on mw2079 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:19] PROBLEM - puppet last run on mw2096 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:19] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:19] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:27] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:27] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:28] PROBLEM - puppet last run on mw2070 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:28] PROBLEM - puppet last run on db2047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:29] PROBLEM - puppet last run on mw2212 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:37] PROBLEM - puppet last run on mw2003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:38] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:48] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:49] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:49] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:57] PROBLEM - puppet last run on mw2093 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:58] PROBLEM - puppet last run on mw2090 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:58] PROBLEM - puppet last run on wtp2010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:58] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:58] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:58] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:58] PROBLEM - puppet last run on mw2123 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:59] PROBLEM - puppet last run on mw2047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:17:59] PROBLEM - puppet last run on mw2056 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:07] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:07] PROBLEM - puppet last run on mw2143 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:08] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:08] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:08] PROBLEM - puppet last run on mw2084 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:08] PROBLEM - puppet last run on mw2113 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:08] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:09] PROBLEM - puppet last run on mw2127 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:09] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:18] PROBLEM - puppet last run on mw2039 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:18] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:18] PROBLEM - puppet last run on wtp2016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:19] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:19] PROBLEM - puppet last run on mw2134 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:19] PROBLEM - puppet last run on mw2030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:19] PROBLEM - puppet last run on mw2055 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:27] PROBLEM - puppet last run on mw2176 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:27] PROBLEM - puppet last run on mw2196 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:27] PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:27] PROBLEM - puppet last run on mw2067 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:29] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:29] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:38] PROBLEM - puppet last run on analytics1056 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:38] PROBLEM - puppet last run on mw2142 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:38] PROBLEM - puppet last run on mw2184 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:39] PROBLEM - puppet last run on nembus is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:39] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:39] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:47] PROBLEM - puppet last run on mw2049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:48] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:48] PROBLEM - puppet last run on mw2131 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:49] PROBLEM - puppet last run on analytics1051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:49] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:18:57] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:09] PROBLEM - puppet last run on mw2101 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:09] PROBLEM - puppet last run on mw2110 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:09] PROBLEM - puppet last run on ms-be1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:17] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:17] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:18] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:18] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:18] PROBLEM - puppet last run on mw2168 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:18] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:28] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:28] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:28] PROBLEM - puppet last run on analytics1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:32] <_joe_> wtf? [06:19:39] PROBLEM - puppet last run on mw2092 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:39] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:48] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:48] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:48] PROBLEM - puppet last run on mw2130 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:49] PROBLEM - puppet last run on mw2062 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:58] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:58] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:58] PROBLEM - puppet last run on mw2098 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:58] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:58] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:59] PROBLEM - puppet last run on mw2203 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:19:59] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:07] PROBLEM - puppet last run on mw2152 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:07] PROBLEM - puppet last run on analytics1052 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:18] PROBLEM - puppet last run on mw2200 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:18] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:28] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:28] PROBLEM - puppet last run on mw2111 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:28] PROBLEM - puppet last run on mw2085 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:29] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:38] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:38] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:39] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:47] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:47] PROBLEM - puppet last run on analytics1044 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:47] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:48] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:48] PROBLEM - puppet last run on wtp2009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:48] PROBLEM - puppet last run on labtestmetal2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:58] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:58] PROBLEM - puppet last run on mw2107 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:20:59] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:07] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:08] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:08] PROBLEM - puppet last run on mira is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:09] PROBLEM - puppet last run on mw2094 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:09] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:09] PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:18] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:27] hrm [06:21:29] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:38] PROBLEM - puppet last run on ms-be2014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:39] PROBLEM - puppet last run on mw2174 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:39] PROBLEM - puppet last run on mw2053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:47] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:48] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:48] PROBLEM - puppet last run on mw2150 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:57] PROBLEM - puppet last run on mw2133 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:57] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:57] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:57] PROBLEM - puppet last run on mw2027 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:57] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:21:58] PROBLEM - puppet last run on ms-be1017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:07] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:18] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:18] PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:27] PROBLEM - puppet last run on labvirt1005 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:27] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:28] PROBLEM - puppet last run on mw2048 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:28] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:29] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:47] PROBLEM - puppet last run on labvirt1007 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:48] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:49] PROBLEM - puppet last run on mw2046 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:49] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:49] PROBLEM - puppet last run on labcontrol1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:57] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:58] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:22:58] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:07] PROBLEM - puppet last run on ms-be2016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:08] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:08] PROBLEM - puppet last run on mw2202 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:08] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:09] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:18] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:18] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:19] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:28] PROBLEM - puppet last run on mw2026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:37] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:37] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:37] PROBLEM - puppet last run on mw2186 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:38] PROBLEM - puppet last run on ms-be2017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:38] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:39] PROBLEM - puppet last run on mw2106 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:47] PROBLEM - puppet last run on mw2068 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:48] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:48] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:48] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:48] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:57] PROBLEM - puppet last run on db2050 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:58] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:23:59] PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:07] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:09] PROBLEM - puppet last run on labservices1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:09] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:17] PROBLEM - puppet last run on mw2091 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:18] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:18] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:18] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:18] PROBLEM - puppet last run on mw2167 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:18] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:18] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:27] PROBLEM - puppet last run on ms-be1019 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:27] PROBLEM - puppet last run on es2005 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:27] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:28] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:28] PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:28] PROBLEM - puppet last run on mw2058 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:28] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:37] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:37] PROBLEM - puppet last run on mw2032 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:38] PROBLEM - puppet last run on mw2193 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:47] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:47] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:48] PROBLEM - puppet last run on mw2008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:48] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:49] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:49] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:57] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:58] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:24:58] PROBLEM - puppet last run on labvirt1004 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:07] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:08] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:09] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:17] PROBLEM - puppet last run on labtestcontrol2001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:17] PROBLEM - puppet last run on dbproxy1008 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:18] PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:18] PROBLEM - puppet last run on mw2209 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:28] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:28] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:28] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:28] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:29] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:29] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:29] PROBLEM - puppet last run on es2009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:30] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:37] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:37] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:37] PROBLEM - puppet last run on mw2057 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:38] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:38] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:38] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:39] PROBLEM - puppet last run on ms-be2020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:39] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:39] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:47] PROBLEM - puppet last run on mw2157 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:47] PROBLEM - puppet last run on mw2076 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:48] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:50] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:50] PROBLEM - puppet last run on mw2023 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:57] PROBLEM - puppet last run on mw1260 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:57] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:57] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:58] PROBLEM - puppet last run on mw2141 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:59] PROBLEM - puppet last run on mw2118 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:25:59] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:07] PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:07] PROBLEM - puppet last run on mw2020 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:08] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:08] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:09] PROBLEM - puppet last run on mw2021 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:09] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:18] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:18] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:18] PROBLEM - puppet last run on mw2069 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:26:35] <_joe_> this is all somewhat bogus, no reason to be too alarmed [06:27:05] killed the icinga-wm . it will come back in a while [06:27:42] <_joe_> ok [06:31:08] PROBLEM - puppet last run on mw2132 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:08] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:09] PROBLEM - puppet last run on mw2181 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:18] PROBLEM - puppet last run on mw2088 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:18] PROBLEM - puppet last run on db2049 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:18] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:27] PROBLEM - puppet last run on mw2054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:28] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:28] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:29] PROBLEM - puppet last run on mw2037 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:29] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:37] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:38] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:38] PROBLEM - puppet last run on mw2012 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:39] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:47] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:47] PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:47] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:31:47] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:47] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:48] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:48] PROBLEM - puppet last run on mw2125 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:48] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:48] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:58] PROBLEM - puppet last run on db2051 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:58] PROBLEM - puppet last run on mw2072 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:59] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:31:59] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [06:39:19] (03CR) 10Krinkle: Tidy robots.txt (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/240065 (https://phabricator.wikimedia.org/T104251) (owner: 10Mdann52) [06:58:33] <_joe_> !log trying to manually syncing the ubuntu mirrors, since they are corrupted at the moment [06:58:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [07:02:49] 6operations: uwsgi takes a long time to restart (Debian Jessie in labs) - https://phabricator.wikimedia.org/T118495#1803094 (10yuvipanda) a:5yuvipanda>3None [07:06:58] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: puppet fail [07:13:18] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:13:37] RECOVERY - puppet last run on analytics1048 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:14:28] RECOVERY - puppet last run on analytics1057 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:14:29] RECOVERY - puppet last run on wtp2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:14:39] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:14:48] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:14:48] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:14:57] RECOVERY - puppet last run on wtp2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:15:27] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:15:29] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:15:37] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:15:48] RECOVERY - puppet last run on mw2080 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:15:48] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:15:49] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:15:49] RECOVERY - puppet last run on labtestnet2001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:15:57] RECOVERY - puppet last run on mw2015 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:15:59] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:07] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:08] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:16:08] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:08] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:08] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:16:08] RECOVERY - puppet last run on mw2188 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:16:08] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:16:09] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:17] RECOVERY - puppet last run on db2045 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:16:18] RECOVERY - puppet last run on db2054 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:16:18] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:16:18] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:16:19] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:19] RECOVERY - puppet last run on wtp2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:27] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:28] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:28] RECOVERY - puppet last run on mw2109 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:16:29] RECOVERY - puppet last run on mw2114 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:16:29] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:29] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:16:37] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:38] RECOVERY - puppet last run on labvirt1009 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:16:47] RECOVERY - puppet last run on labcontrol1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:16:48] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:16:49] RECOVERY - puppet last run on mw2105 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:57] RECOVERY - puppet last run on mw2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:57] RECOVERY - puppet last run on es2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:59] RECOVERY - puppet last run on db2047 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:17:07] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:17:08] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:17:08] RECOVERY - puppet last run on mw2212 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:17:08] RECOVERY - puppet last run on mw2163 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:17:18] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:18] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:17:28] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:17:28] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:28] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:17:28] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:17:29] RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:29] RECOVERY - puppet last run on mw2093 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:17:37] RECOVERY - puppet last run on mw2082 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:37] RECOVERY - puppet last run on wtp2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:38] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:17:38] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:38] RECOVERY - puppet last run on mw2123 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:17:38] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:17:39] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:17:39] RECOVERY - puppet last run on mw2143 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:17:39] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:47] RECOVERY - puppet last run on mw2113 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:17:48] RECOVERY - puppet last run on mw2127 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:17:48] RECOVERY - puppet last run on mw2087 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:17:57] RECOVERY - puppet last run on mw2039 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:17:57] RECOVERY - puppet last run on wtp2016 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:17:58] RECOVERY - puppet last run on mw2134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:58] RECOVERY - puppet last run on mw2117 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:58] RECOVERY - puppet last run on mw2196 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:17:58] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:17:59] RECOVERY - puppet last run on mw2176 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:17:59] RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:17:59] RECOVERY - puppet last run on mw2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:00] RECOVERY - puppet last run on mw2067 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:18:00] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:18:01] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:18:01] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:18:08] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:18:08] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:08] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:08] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:09] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:18:09] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:18:17] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:18] RECOVERY - puppet last run on mw2142 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:18:19] RECOVERY - puppet last run on mw2184 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:18:19] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:19] RECOVERY - puppet last run on nembus is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:18:19] RECOVERY - puppet last run on analytics1051 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:18:19] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:19] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:18:19] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:20] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:18:27] RECOVERY - puppet last run on mw2131 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:18:27] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:28] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:28] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:28] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:18:29] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:38] RECOVERY - puppet last run on ms-be1021 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:18:38] RECOVERY - puppet last run on mw2083 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:47] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:47] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:18:48] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:48] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:18:48] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:18:48] RECOVERY - puppet last run on mw2101 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:18:48] RECOVERY - puppet last run on mw2079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:49] RECOVERY - puppet last run on mw2096 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:49] RECOVERY - puppet last run on mw2110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:50] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:50] RECOVERY - puppet last run on mw2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:57] RECOVERY - puppet last run on mw2070 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:18:57] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:18:58] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:18:58] RECOVERY - puppet last run on analytics1002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:18:59] RECOVERY - puppet last run on mw2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:08] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:09] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:09] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:18] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:19:18] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:18] RECOVERY - puppet last run on mw2092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:19] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:27] RECOVERY - puppet last run on mw2090 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:28] RECOVERY - puppet last run on mw2130 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:19:28] RECOVERY - puppet last run on mw2047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:29] RECOVERY - puppet last run on mw2062 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:19:29] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:29] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:29] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:19:29] RECOVERY - puppet last run on mw2056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:29] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:19:37] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:19:37] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:37] RECOVERY - puppet last run on mw2084 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:37] RECOVERY - puppet last run on mw2098 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:19:37] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:19:38] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:38] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:39] RECOVERY - puppet last run on mw2203 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:19:39] RECOVERY - puppet last run on mw2152 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:19:48] RECOVERY - puppet last run on mw2030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:48] RECOVERY - puppet last run on mw2055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:48] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:49] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:49] RECOVERY - puppet last run on mw2200 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:19:58] RECOVERY - puppet last run on analytics1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:19:58] RECOVERY - Router interfaces on cr2-codfw is OK: OK: host 208.80.153.193, interfaces up: 119, down: 0, dormant: 0, excluded: 1, unused: 0 [07:20:07] RECOVERY - puppet last run on mw2111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:08] RECOVERY - puppet last run on mw2085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:08] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:20:08] RECOVERY - puppet last run on mw2049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:08] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:20:09] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:20:17] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:17] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:20:19] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:27] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:20:28] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:39] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:39] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:20:39] RECOVERY - puppet last run on mw2168 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:47] RECOVERY - puppet last run on mira is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:48] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:48] RECOVERY - puppet last run on mw2172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:48] RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:20:59] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:21:08] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:18] RECOVERY - puppet last run on mw2174 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:18] RECOVERY - puppet last run on ms-be2014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:19] RECOVERY - puppet last run on mw2053 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:21:27] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:27] RECOVERY - puppet last run on mw2106 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:21:28] RECOVERY - puppet last run on analytics1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:28] RECOVERY - puppet last run on mw2150 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:21:29] RECOVERY - puppet last run on mw2133 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:21:37] RECOVERY - puppet last run on mw2027 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:21:38] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:49] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:57] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:21:58] RECOVERY - puppet last run on labvirt1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:58] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:21:59] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:07] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:07] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:22:08] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:22:09] RECOVERY - puppet last run on mw2048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:09] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:09] RECOVERY - puppet last run on analytics1044 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:17] RECOVERY - puppet last run on labvirt1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:18] RECOVERY - puppet last run on wtp2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:19] RECOVERY - puppet last run on labtestmetal2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:27] RECOVERY - puppet last run on mw2193 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:22:27] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:22:27] RECOVERY - puppet last run on labcontrol1002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:22:27] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:22:28] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:22:28] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:22:28] RECOVERY - puppet last run on mw2107 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:29] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:22:29] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:30] RECOVERY - puppet last run on mw2046 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:22:38] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:22:38] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:38] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:39] RECOVERY - puppet last run on ms-be2016 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:22:47] RECOVERY - puppet last run on mw2202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:47] RECOVERY - puppet last run on mw2094 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:48] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:49] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:23:07] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:08] RECOVERY - puppet last run on mw2026 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:23:08] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:17] RECOVERY - puppet last run on mw2186 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:23:17] RECOVERY - puppet last run on ms-be2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:18] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:23:18] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:18] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:19] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:19] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:23:19] RECOVERY - puppet last run on ms-be1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:27] RECOVERY - puppet last run on mw2068 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:23:28] RECOVERY - puppet last run on db2050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:38] RECOVERY - puppet last run on mw2144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:39] RECOVERY - puppet last run on labservices1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:23:48] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:48] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:48] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:48] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:48] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:23:49] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:23:49] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:23:57] RECOVERY - puppet last run on mw2167 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:57] RECOVERY - puppet last run on ms-be1019 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:23:58] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:23:59] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:23:59] RECOVERY - puppet last run on es2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:08] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:24:08] RECOVERY - puppet last run on mw2058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:17] RECOVERY - puppet last run on mw2032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:18] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:18] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:24:19] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:24:19] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:24:27] RECOVERY - puppet last run on mw2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:28] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:28] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:29] RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:39] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:47] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:47] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:24:48] RECOVERY - puppet last run on dbproxy1008 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:24:49] RECOVERY - puppet last run on labtestcontrol2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:49] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:24:58] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:24:58] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:58] RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:58] RECOVERY - puppet last run on mw2209 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:24:58] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:24:58] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:24:59] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:25:07] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:25:08] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:25:08] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:08] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:08] RECOVERY - puppet last run on es2009 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:25:08] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:25:09] RECOVERY - puppet last run on mw2057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:17] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:25:17] RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:25:18] RECOVERY - puppet last run on ms-be2020 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:25:18] RECOVERY - puppet last run on mw2157 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:25:19] RECOVERY - puppet last run on mw2076 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:25:19] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:25:19] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:25:19] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:20] RECOVERY - puppet last run on mw1260 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:25:27] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:25:27] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:28] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:37] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:25:38] RECOVERY - puppet last run on mw2141 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:25:38] RECOVERY - puppet last run on mw2118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:38] RECOVERY - puppet last run on wtp2017 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:25:39] RECOVERY - puppet last run on mw2091 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:39] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:39] RECOVERY - puppet last run on mw2020 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:25:39] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:25:48] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:25:48] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:25:48] RECOVERY - puppet last run on mw2021 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:25:48] RECOVERY - puppet last run on analytics1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:57] RECOVERY - puppet last run on mw2069 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:26:07] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:26:07] RECOVERY - puppet last run on mw2145 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:26:08] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:26:08] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:09] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:09] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:26:09] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:26:09] RECOVERY - puppet last run on labnet1002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:26:18] RECOVERY - puppet last run on mw2081 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:26:18] RECOVERY - puppet last run on labvirt1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:27] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:28] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:26:28] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:26:29] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:26:37] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:26:37] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:26:38] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:26:38] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:26:48] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:26:48] RECOVERY - puppet last run on mw2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:48] RECOVERY - puppet last run on mw2033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:57] RECOVERY - puppet last run on wtp2015 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:26:57] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:26:58] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:58] RECOVERY - puppet last run on subra is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:58] RECOVERY - puppet last run on mw2016 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:26:58] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:26:58] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:27:07] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:18] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:27:18] RECOVERY - puppet last run on wtp2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:18] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:19] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:20] RECOVERY - puppet last run on mw2023 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:27:38] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:27:38] RECOVERY - puppet last run on db2044 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:27:48] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:27:48] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:57] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:27:58] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:27:58] RECOVERY - puppet last run on mw2095 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:27:58] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:28:07] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:28:07] RECOVERY - puppet last run on mw2052 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:28:07] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:28:08] RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:08] RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:28:08] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:28:08] RECOVERY - puppet last run on labtestvirt2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:17] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:18] RECOVERY - puppet last run on mw2043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:19] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:28:19] RECOVERY - puppet last run on eventlog2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:19] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:28:19] RECOVERY - puppet last run on mw2050 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:28:39] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:28:40] RECOVERY - puppet last run on mw2120 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:28:47] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:28:48] RECOVERY - puppet last run on ms-be2021 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:28:48] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:48] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:48] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:49] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:28:57] RECOVERY - puppet last run on labtestneutron2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:58] RECOVERY - puppet last run on wtp2004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:28:58] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:58] RECOVERY - puppet last run on mw2077 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:07] RECOVERY - puppet last run on suhail is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:29:07] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:29:07] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:29:07] RECOVERY - puppet last run on mw2138 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:29:08] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:29:09] RECOVERY - puppet last run on dbproxy1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:29:09] RECOVERY - puppet last run on wtp2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:09] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:17] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:29:18] RECOVERY - puppet last run on mw2024 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:29:18] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:29:28] RECOVERY - puppet last run on mw2177 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:29:28] RECOVERY - puppet last run on mw2198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:28] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:29:29] RECOVERY - puppet last run on mw2136 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:37] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:38] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:48] RECOVERY - puppet last run on mw2146 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:57] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:57] RECOVERY - puppet last run on mw2166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:58] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:29:58] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:29:58] RECOVERY - puppet last run on mw2061 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:29:58] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:29:58] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:07] RECOVERY - puppet last run on graphite2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:08] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:17] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:19] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:19] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:19] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:19] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:27] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:30:27] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:27] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:30:28] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:30:28] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:28] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:37] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:38] RECOVERY - puppet last run on es2007 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:30:38] RECOVERY - puppet last run on mw2132 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:30:47] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:30:48] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:30:48] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:48] RECOVERY - puppet last run on mw2181 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:30:48] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:30:49] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:49] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:30:58] RECOVERY - puppet last run on mw2088 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:58] RECOVERY - puppet last run on db2049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:58] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:58] RECOVERY - puppet last run on analytics1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:58] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:58] RECOVERY - puppet last run on mw2054 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:31:07] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:31:08] RECOVERY - puppet last run on mw2037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:08] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:31:08] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:09] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:31:17] RECOVERY - puppet last run on mw1259 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:17] RECOVERY - puppet last run on wtp2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:17] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:31:18] RECOVERY - puppet last run on mw2012 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:31:18] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:18] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:31:18] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:18] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:19] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:31:19] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:31:28] RECOVERY - puppet last run on mw2214 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:31:28] RECOVERY - puppet last run on mw2125 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:31:28] RECOVERY - puppet last run on mw2035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:28] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:31:29] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:37] RECOVERY - puppet last run on db2051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:37] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:37] RECOVERY - puppet last run on mw2072 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:31:37] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:31:38] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:31:48] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:49] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:31:49] RECOVERY - puppet last run on analytics1054 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:31:49] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:31:49] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:49] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:49] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:57] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:57] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:58] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:58] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:31:58] RECOVERY - puppet last run on mw2162 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:31:59] RECOVERY - puppet last run on mw2191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:31:59] RECOVERY - puppet last run on ms-be1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:08] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:32:08] RECOVERY - puppet last run on mw2169 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:32:09] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:32:17] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:32:17] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:27] RECOVERY - puppet last run on analytics1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:28] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:38] RECOVERY - puppet last run on mw2187 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:32:38] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:38] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:32:48] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:32:48] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:49] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:32:58] RECOVERY - puppet last run on mw2038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:58] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:58] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:32:58] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:59] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:07] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:33:07] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:33:07] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:07] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:33:07] RECOVERY - puppet last run on analytics1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:08] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:08] RECOVERY - puppet last run on graphite1001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:33:17] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:33:19] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:33:27] RECOVERY - puppet last run on mw2074 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:33:28] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:28] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:33:37] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:33:38] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:38] RECOVERY - puppet last run on mw2028 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:33:47] RECOVERY - puppet last run on labvirt1008 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:33:47] RECOVERY - puppet last run on mw2051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:48] RECOVERY - puppet last run on wtp2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:48] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:49] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:33:49] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:33:49] RECOVERY - puppet last run on mw2122 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:33:58] RECOVERY - puppet last run on mw2155 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:58] RECOVERY - puppet last run on db2053 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:33:58] RECOVERY - puppet last run on labvirt1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:33:58] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:07] RECOVERY - puppet last run on mw2159 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:34:07] RECOVERY - puppet last run on mw2175 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:34:08] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:34:08] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:34:17] RECOVERY - puppet last run on analytics1055 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:34:17] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:34:19] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:19] RECOVERY - puppet last run on mw2108 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:34:27] RECOVERY - puppet last run on mw2210 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:28] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:34:28] RECOVERY - puppet last run on mw2205 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:34:28] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:29] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:38] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:34:47] RECOVERY - puppet last run on dbstore2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:49] RECOVERY - puppet last run on mw2185 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:34:49] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:34:49] RECOVERY - puppet last run on wtp2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:49] RECOVERY - puppet last run on mw2121 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:34:49] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:57] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:57] RECOVERY - puppet last run on mw2034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:58] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:58] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:59] RECOVERY - puppet last run on dbstore2002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:35:07] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:35:08] RECOVERY - puppet last run on wtp2018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:35:08] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:35:08] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:09] RECOVERY - puppet last run on mw2014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:09] RECOVERY - puppet last run on mw2213 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:35:09] RECOVERY - puppet last run on mw2071 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:09] RECOVERY - puppet last run on mw2180 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:35:17] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:35:18] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:35:18] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:18] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:27] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:35:29] RECOVERY - puppet last run on mw2148 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:35:37] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:38] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:35:38] RECOVERY - puppet last run on mw2204 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:38] RECOVERY - puppet last run on mw2154 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:39] RECOVERY - puppet last run on mw2124 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:48] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:48] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:35:48] RECOVERY - puppet last run on mw2115 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:35:49] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:49] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:35:57] RECOVERY - puppet last run on mw2128 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:35:58] RECOVERY - puppet last run on mw2104 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:35:59] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:36:07] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:36:17] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:36:18] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:18] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:36:28] RECOVERY - puppet last run on db2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:28] RECOVERY - puppet last run on ms-be2018 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:36:28] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:36:28] RECOVERY - puppet last run on mw2013 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:36:28] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:36:29] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:36:37] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:36:37] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:37] RECOVERY - puppet last run on mw2001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:36:38] RECOVERY - puppet last run on mw2156 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:36:39] RECOVERY - puppet last run on mw2066 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:36:47] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:36:48] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:36:48] RECOVERY - puppet last run on labvirt1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:36:57] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:36:57] RECOVERY - puppet last run on mw2063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:57] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:36:58] RECOVERY - puppet last run on labvirt1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:59] RECOVERY - puppet last run on mw2171 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:37:07] RECOVERY - puppet last run on mw2022 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:37:08] RECOVERY - puppet last run on mw2173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:09] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:37:09] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:18] RECOVERY - puppet last run on mw2078 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:37:18] RECOVERY - puppet last run on mw2040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:18] RECOVERY - puppet last run on mw2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:18] RECOVERY - puppet last run on wtp2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:27] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:37:27] RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:37:28] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:28] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:28] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:37:28] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:37] RECOVERY - puppet last run on mw2097 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:37:38] RECOVERY - puppet last run on mw2060 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:37:39] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:39] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:39] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:47] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:37:47] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:37:48] RECOVERY - puppet last run on wtp2012 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:37:48] RECOVERY - puppet last run on mw2137 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:57] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:58] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:38:07] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:18] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:18] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:38:18] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:19] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:19] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:28] RECOVERY - puppet last run on mw2195 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:28] RECOVERY - puppet last run on mw2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:28] RECOVERY - puppet last run on mw2206 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:29] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:38] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:38:38] RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:38] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:38] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:38:38] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:38:47] RECOVERY - puppet last run on labvirt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:48] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:38:48] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:38:49] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:39:08] RECOVERY - puppet last run on analytics1046 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:39:09] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:39:09] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:39:09] RECOVERY - puppet last run on db2043 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:39:18] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:39:18] RECOVERY - puppet last run on mw2086 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:39:18] RECOVERY - puppet last run on mw2119 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:18] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:19] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:39:27] RECOVERY - puppet last run on mw2192 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:39:28] RECOVERY - puppet last run on mw2211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:28] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:39:28] RECOVERY - puppet last run on mw2178 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:39:28] RECOVERY - puppet last run on mw2002 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:39:29] RECOVERY - puppet last run on mw2059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:29] RECOVERY - puppet last run on mw2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:29] RECOVERY - puppet last run on mw2029 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:39:49] RECOVERY - puppet last run on db2048 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:39:49] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:39:58] RECOVERY - puppet last run on ms-be2019 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:40:07] RECOVERY - puppet last run on ms-be1016 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:40:08] RECOVERY - puppet last run on mw2190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:08] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:08] RECOVERY - puppet last run on mw2025 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:40:17] RECOVERY - puppet last run on mw2147 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:40:17] RECOVERY - puppet last run on analytics1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:17] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:40:18] RECOVERY - puppet last run on wtp2014 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:40:27] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:28] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:40:37] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:40:38] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:40:38] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:40:39] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:47] RECOVERY - puppet last run on mw2006 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:40:47] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:40:48] RECOVERY - puppet last run on mw2135 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:49] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:40:49] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:40:57] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:40:58] RECOVERY - puppet last run on es2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:59] RECOVERY - puppet last run on dbproxy1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:07] RECOVERY - puppet last run on mw2149 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:07] RECOVERY - puppet last run on mw2165 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:07] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:41:17] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:17] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:19] RECOVERY - puppet last run on mw2194 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:41:28] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:41:38] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:41:39] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:41:50] RECOVERY - puppet last run on labvirt1011 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:41:57] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:58] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:58] RECOVERY - puppet last run on mw2041 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:41:58] RECOVERY - puppet last run on mw2151 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:41:58] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:41:58] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:41:59] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:08] RECOVERY - puppet last run on mw2089 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:42:08] RECOVERY - puppet last run on mw2103 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:42:08] RECOVERY - puppet last run on mw2102 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:42:17] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:17] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:42:17] RECOVERY - puppet last run on ms-be1018 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:42:17] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:42:18] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:42:18] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:42:18] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:42:20] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:27] RECOVERY - puppet last run on mw2112 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:42:27] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:42:27] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:28] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:42:28] RECOVERY - puppet last run on mw2042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:29] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:42:37] RECOVERY - puppet last run on wtp2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:37] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:38] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:42:38] RECOVERY - puppet last run on mw2099 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:42:47] RECOVERY - puppet last run on mw2197 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:42:48] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:42:49] RECOVERY - puppet last run on eventlog1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:57] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:42:57] RECOVERY - puppet last run on mw2139 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:42:57] RECOVERY - puppet last run on mw2179 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:57] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:58] RECOVERY - puppet last run on mw2160 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:42:58] RECOVERY - puppet last run on mw2183 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:42:58] RECOVERY - puppet last run on mw2170 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:42:59] RECOVERY - puppet last run on mw2189 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:59] RECOVERY - puppet last run on mw2065 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:43:00] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:00] RECOVERY - puppet last run on mw2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:01] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:01] RECOVERY - puppet last run on mw2201 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:43:07] RECOVERY - puppet last run on wtp2002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:43:07] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:07] RECOVERY - puppet last run on db2046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:07] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:09] RECOVERY - puppet last run on mw2044 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:43:18] RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:18] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:43:28] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:43:28] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:38] RECOVERY - puppet last run on californium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:38] RECOVERY - puppet last run on mw2140 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:47] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:43:48] RECOVERY - puppet last run on mw2164 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:43:48] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:43:57] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:43:58] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:58] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:59] RECOVERY - puppet last run on mw2116 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:08] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:08] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:08] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:09] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:17] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:18] RECOVERY - puppet last run on mw2100 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:44:28] RECOVERY - puppet last run on analytics1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:28] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:44:37] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:37] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:38] RECOVERY - puppet last run on mw2064 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:44:38] RECOVERY - puppet last run on mw2009 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:44:38] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:44:47] RECOVERY - puppet last run on mw2031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:58] RECOVERY - puppet last run on mw2153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:44:59] RECOVERY - puppet last run on mw2161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:45:08] RECOVERY - puppet last run on mw2199 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:45:09] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:45:17] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:45:18] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:45:27] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:45:39] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:57:59] PROBLEM - puppet last run on mw2081 is CRITICAL: CRITICAL: puppet fail [08:01:38] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 1 below the confidence bounds [08:02:49] mw fixed [08:03:48] RECOVERY - puppet last run on mw2081 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:05:04] I only see the known thumbnail path issue on uploads, nothing else going on right now [08:12:11] YuviPanda: anymore resets today ? :) i flooded my email :D [08:16:18] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 2 below the confidence bounds [08:20:18] (03PS1) 10Jcrespo: Repool db1035 at 100%, depool db1015 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252898 [08:21:48] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 4 below the confidence bounds [08:33:07] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 13 data above and 1 below the confidence bounds [08:34:40] (03CR) 10Jcrespo: [C: 032] Repool db1035 at 100%, depool db1015 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252898 (owner: 10Jcrespo) [08:36:47] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 4 below the confidence bounds [08:37:17] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1035 at 100%, depool db1015 (duration: 01m 31s) [08:37:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:38:09] some host failed [08:38:39] tin failed [08:51:17] PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [5000000.0] [08:51:28] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 4 below the confidence bounds [08:54:37] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [5000000.0] [08:58:18] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [5000000.0] [08:58:58] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 1 below the confidence bounds [09:00:53] (03CR) 10Alexandros Kosiaris: [C: 032] Updated RuboCop to the newest released version [puppet] - 10https://gerrit.wikimedia.org/r/252721 (https://phabricator.wikimedia.org/T112651) (owner: 10Zfilipin) [09:01:02] (03PS2) 10Alexandros Kosiaris: Updated RuboCop to the newest released version [puppet] - 10https://gerrit.wikimedia.org/r/252721 (https://phabricator.wikimedia.org/T112651) (owner: 10Zfilipin) [09:03:59] PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [5000000.0] [09:06:00] (03CR) 10Alexandros Kosiaris: [V: 032] Updated RuboCop to the newest released version [puppet] - 10https://gerrit.wikimedia.org/r/252721 (https://phabricator.wikimedia.org/T112651) (owner: 10Zfilipin) [09:08:08] RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 1.00% above the threshold [1000000.0] [09:08:51] (03CR) 10Alexandros Kosiaris: [C: 032] Restrict LDAP access on the corp LDAP mirror [puppet] - 10https://gerrit.wikimedia.org/r/252730 (owner: 10Muehlenhoff) [09:09:49] (03PS2) 10Alexandros Kosiaris: Restrict LDAP access on the corp LDAP mirror [puppet] - 10https://gerrit.wikimedia.org/r/252730 (owner: 10Muehlenhoff) [09:11:28] RECOVERY - Kafka Broker Replica Max Lag on kafka1020 is OK: OK: Less than 1.00% above the threshold [1000000.0] [09:12:07] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 1 below the confidence bounds [09:20:40] !log performing table optimization on db1015, expect some lag while depooled [09:20:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:29:38] (03PS4) 10Yuvipanda: admin: Add new key for self (ori) [puppet] - 10https://gerrit.wikimedia.org/r/252866 (owner: 10Ori.livneh) [09:29:48] (03CR) 10Yuvipanda: [C: 032 V: 032] admin: Add new key for self (ori) [puppet] - 10https://gerrit.wikimedia.org/r/252866 (owner: 10Ori.livneh) [09:34:37] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [09:36:10] 6operations, 6Analytics-Backlog, 10Analytics-Cluster: Audit Hadoop worker memory usage. - https://phabricator.wikimedia.org/T118501#1803197 (10JAllemandou) @Ottomata: Let me know when you want to have a go on that, I'm interested in helping :) [09:45:16] (03PS2) 10Merlijn van Deen: toollabs: Don't explicitly declare redis collector [puppet] - 10https://gerrit.wikimedia.org/r/252859 (owner: 10Yuvipanda) [09:45:33] (03CR) 10Merlijn van Deen: [C: 031] toollabs: Don't explicitly declare redis collector [puppet] - 10https://gerrit.wikimedia.org/r/252859 (owner: 10Yuvipanda) [09:54:08] (03CR) 10Hashar: "Since renaming from 'rakefile' to 'Rakefile' causes rubocop to catch it, maybe we can do the rename and fix the rubocop issues in a differ" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/252686 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin) [09:57:15] (03CR) 10Giuseppe Lavagetto: [C: 032] toollabs: Don't explicitly declare redis collector [puppet] - 10https://gerrit.wikimedia.org/r/252859 (owner: 10Yuvipanda) [10:06:05] (03PS1) 10Hashar: Rename /rakefile -> /Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252904 [10:06:07] (03PS1) 10Hashar: Retab /Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252905 [10:06:09] (03PS1) 10Hashar: Rakefile: fix a bunch of linting issues [puppet] - 10https://gerrit.wikimedia.org/r/252906 [10:06:37] (03CR) 10Hashar: "Renamed rakefile to Rakefile, retabbed it and fixed all rubocop issues with:" [puppet] - 10https://gerrit.wikimedia.org/r/252686 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin) [10:25:29] good morning [10:25:35] hashar, one thing [10:25:43] I did a deploy earlier [10:25:58] and it failed to sync to the master, I think mira failed [10:26:16] you have the error on the logs [10:26:29] on production right ? [10:26:30] are you familiar with scap? Sorry, I am not [10:26:34] yes [10:26:41] i might be a bit more familliar [10:26:45] you have the log on tin [10:26:54] but have little knowledge of all the other systems scap relies on [10:26:55] but ping me otherwise [10:27:03] have you filled a task with traces etc ? [10:27:07] I can handle the systems :-) [10:27:16] no, but I will [10:27:27] I think logs are sent to logstash, will dig there [10:27:49] do you have time frame? [10:27:52] window [10:28:01] whatever my english is crap again [10:30:15] rsync returned non-zero exit status 23 .... [10:30:23] the error handling in scap is crap [10:30:41] it is ok, I will create a ticket with the whole output [10:30:48] "Partial transfer due to error" [10:30:50] what I wanted to make sure [10:31:00] is that someone on releng was in the loop [10:31:17] I don't understand why scap does not abort on failures [10:31:18] (even if I do not think there are deployments for today [10:31:27] (03PS9) 10Giuseppe Lavagetto: Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [10:31:29] yep, my answer, too :-) [10:32:59] <_joe_> oh shit damn submodules [10:33:00] <_joe_> damn [10:33:15] maybe mira is not ready yet :( [10:34:21] _joe_, I was going to say that [10:34:23] (03PS10) 10Giuseppe Lavagetto: Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [10:34:24] :-) [10:35:05] next job is to get rid of it, by public demand [10:35:21] * _joe_ waves fist at otto [10:36:26] he actually said he was ok with it [10:36:41] but I will coordinate fully again with it [10:36:45] *him [10:37:22] (03PS1) 10Zhuyifei1999: Add *.mil to server-side upload whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252908 (https://phabricator.wikimedia.org/T118554) [10:40:00] <_joe_> jynus: I'm running the compiler now [10:40:17] hashar, to be clear, I do not need a fix- the sync worked, creating the ticket now [10:40:29] (I do not need a fix now) [10:43:03] _joe_, I think the patch is ok, what I do not have trust on is on unpuppetized dependencies, specific configuration, etc. [10:43:18] so it may take a couple of trials [10:43:21] <_joe_> jynus: fixing some small issues [10:44:32] perl-mysql is a dependency too, not sure if it can be assumed? [10:45:39] libdbd-mysql is the exact name [10:46:16] sorry, it is libdbd-mysql-perl [10:47:55] (03PS11) 10Giuseppe Lavagetto: Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [10:48:15] <_joe_> jynus: run aptitude why libdbd-mysql-perl [10:48:24] <_joe_> that would tell you if it's already installed [10:49:11] no need: mysql-client-5.5 Depends libdbd-mysql-perl [10:50:29] <_joe_> ok :) [10:53:16] (03CR) 10jenkins-bot: [V: 04-1] Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [10:53:57] (03PS12) 10Giuseppe Lavagetto: Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [10:54:53] (03CR) 10jenkins-bot: [V: 04-1] Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [10:55:16] pff [10:55:31] oh that is "just" puppet-lint [10:55:41] actually I should find a way to make that job faster [10:55:49] by only linting files changed by HEAD [10:56:24] <_joe_> hashar: yeah a lame retab [10:56:37] <_joe_> hashar: for now I am still fixing errors in the compiler [10:56:57] oh speaking of it [10:57:25] when you have a few dependent changes that are pending merging like A -> B -> C [10:57:29] <_joe_> jynus: seems cool https://puppet-compiler.wmflabs.org/1276/ [10:57:35] would running compiler against C includes the parent patches as well? [10:57:38] i.e. does it checkout C [10:57:39] <_joe_> hashar: yeah that's an open problem [10:57:44] or cherry pick C on tip of production branch [10:57:55] <_joe_> we do cherry-pick right now [10:58:02] <_joe_> and I don't want to do a checkout [10:58:16] <_joe_> that is wrong and will cause all sort of false positives [10:58:26] <_joe_> hashar: write a bug, it's not a 10-minutes fix [10:58:59] also https://phabricator.wikimedia.org/T118406 [10:59:38] more subrepo evil :-) [10:59:45] <_joe_> argh [11:00:09] <_joe_> "subrepo changes are not supported" Open > Invalid :P [11:00:22] im ok with that [11:00:53] but I needed that after I scratched my head for days [11:02:39] <_joe_> jynus: yeah I can imagine, sorry [11:02:50] _joe_: I was really just wonderig [11:02:56] no sorry accepted [11:03:07] you have savedme otherweise hundreds of hours [11:03:11] and yeah you would need to do the puppet compile with HEAD^ instead of production :-( [11:03:16] so thanks! [11:04:29] (03CR) 10Dereckson: "Technically, looks good to me." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252908 (https://phabricator.wikimedia.org/T118554) (owner: 10Zhuyifei1999) [11:08:03] 6operations, 5Continuous-Integration-Scaling, 5Patch-For-Review: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1803288 (10hashar) [11:08:23] pff [11:08:31] gallium takes ages to do any file ops [11:14:46] Not sure if banners have improved a lot or gotten broken: https://en.wiktionary.org/wiki/Wiktionary:Main_Page [11:15:15] I would say the first [11:28:58] (03PS1) 10Dereckson: Namespace configuration on ru.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252911 (https://phabricator.wikimedia.org/T117857) [11:29:49] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 9.09% of data above the critical threshold [500.0] [11:33:28] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 3 below the confidence bounds [11:40:44] (03PS13) 10Giuseppe Lavagetto: Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [11:42:39] !log restarting db2060 mysql for upgrade [11:42:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [11:43:03] (03PS2) 10Muehlenhoff: Enable ferm on db2010/db2030 [puppet] - 10https://gerrit.wikimedia.org/r/240055 [11:43:20] (03CR) 10Muehlenhoff: [C: 032 V: 032] Enable ferm on db2010/db2030 [puppet] - 10https://gerrit.wikimedia.org/r/240055 (owner: 10Muehlenhoff) [11:44:47] (03PS14) 10Giuseppe Lavagetto: Adding tendril::maintenance class [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [11:45:11] (03CR) 10Steinsplitter: [C: 031] "Looks good to me." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252908 (https://phabricator.wikimedia.org/T118554) (owner: 10Zhuyifei1999) [11:45:42] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/252208 (owner: 10Jcrespo) [11:46:39] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [11:49:36] (03PS1) 10Giuseppe Lavagetto: tendril::maintenance: fix dependency [puppet] - 10https://gerrit.wikimedia.org/r/252914 [11:50:35] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/252914 (owner: 10Giuseppe Lavagetto) [11:52:47] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: puppet fail [11:54:22] (03PS1) 10Giuseppe Lavagetto: tendril::maintenance: add missing file [puppet] - 10https://gerrit.wikimedia.org/r/252915 [11:54:38] (03CR) 10Giuseppe Lavagetto: [C: 032] tendril::maintenance: add missing file [puppet] - 10https://gerrit.wikimedia.org/r/252915 (owner: 10Giuseppe Lavagetto) [11:54:56] <_joe_> jenkins: behave [11:55:20] (03PS1) 10Muehlenhoff: Uninstall wpasupplicant [puppet] - 10https://gerrit.wikimedia.org/r/252916 [11:55:35] <_joe_> moritzm: wat? [11:55:39] <_joe_> :P [11:55:51] Ubuntu... [11:56:02] nearly 100 systems [11:56:22] (noticed that when checking on required security updates) [11:58:18] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [12:02:41] (03PS1) 10Giuseppe Lavagetto: tendril::maintenance: fix class parameter [puppet] - 10https://gerrit.wikimedia.org/r/252921 [12:03:08] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/252921 (owner: 10Giuseppe Lavagetto) [12:12:18] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [5000000.0] [12:12:20] (03PS1) 10Giuseppe Lavagetto: tendril::maintenance: write crons on one line. [puppet] - 10https://gerrit.wikimedia.org/r/252923 [12:12:41] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] tendril::maintenance: write crons on one line. [puppet] - 10https://gerrit.wikimedia.org/r/252923 (owner: 10Giuseppe Lavagetto) [12:13:46] (03PS1) 10Muehlenhoff: Various finetuning to server groups [puppet] - 10https://gerrit.wikimedia.org/r/252924 [12:14:35] (03Abandoned) 10Muehlenhoff: Various finetuning to server groups [puppet] - 10https://gerrit.wikimedia.org/r/252924 (owner: 10Muehlenhoff) [12:15:37] (03PS1) 10Muehlenhoff: More finetuning for server groups [puppet] - 10https://gerrit.wikimedia.org/r/252925 [12:17:26] (03Abandoned) 10Hashar: nodepool: send metrics to statsd [puppet] - 10https://gerrit.wikimedia.org/r/237721 (owner: 10Hashar) [12:18:19] (03CR) 10Muehlenhoff: [C: 032 V: 032] More finetuning for server groups [puppet] - 10https://gerrit.wikimedia.org/r/252925 (owner: 10Muehlenhoff) [12:18:31] (03PS2) 10Hashar: contint: setup zuul-merger on scandium.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/252336 (https://phabricator.wikimedia.org/T95046) [12:20:43] (03CR) 10Giuseppe Lavagetto: [C: 04-1] Use a more useful error message when DB connection fails (031 comment) [software/dbtree] - 10https://gerrit.wikimedia.org/r/251791 (owner: 10Alex Monk) [12:21:54] (03PS2) 10Alexandros Kosiaris: Rename /rakefile -> /Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252904 (owner: 10Hashar) [12:22:03] (03CR) 10Alexandros Kosiaris: [C: 032] Rename /rakefile -> /Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252904 (owner: 10Hashar) [12:22:36] (03CR) 10Alexandros Kosiaris: [C: 032] Retab /Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252905 (owner: 10Hashar) [12:22:54] (03CR) 10Hashar: "Puppet compile seems all happy. https://puppet-compiler.wmflabs.org/1277/scandium.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/252336 (https://phabricator.wikimedia.org/T95046) (owner: 10Hashar) [12:27:09] RECOVERY - Kafka Broker Replica Max Lag on kafka1022 is OK: OK: Less than 1.00% above the threshold [1000000.0] [12:29:52] akosiaris: the poor /Rakefile changes needs a few more rebases :-( [12:30:21] hashar: yeah, working on it [12:37:12] (03PS2) 10Alexandros Kosiaris: Retab /Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252905 (owner: 10Hashar) [12:38:14] (03CR) 10Alexandros Kosiaris: [C: 032] Rakefile: fix a bunch of linting issues [puppet] - 10https://gerrit.wikimedia.org/r/252906 (owner: 10Hashar) [12:38:29] (03PS2) 10Alexandros Kosiaris: Rakefile: fix a bunch of linting issues [puppet] - 10https://gerrit.wikimedia.org/r/252906 (owner: 10Hashar) [12:40:48] (03CR) 10Alexandros Kosiaris: [V: 04-1] "Needs manual rebasing" [puppet] - 10https://gerrit.wikimedia.org/r/252686 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin) [12:43:37] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [12:47:28] hashar: done [12:48:32] (03PS3) 10Hashar: Add rubocop and 'test' target to Rakefile [puppet] - 10https://gerrit.wikimedia.org/r/252686 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin) [12:49:01] (03CR) 10Hashar: "Rebased and reworked, namely I have dropped the task option and the comment about mediawiki vagrant." [puppet] - 10https://gerrit.wikimedia.org/r/252686 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin) [12:49:19] akosiaris: thanks! [12:49:25] will let zeljko review the rebase I did [12:50:42] hashar: ok, thanks! [12:55:56] 6operations, 10ops-esams: Power cr2-esams PEM 2/PEM 3 - https://phabricator.wikimedia.org/T118166#1803395 (10mark) p:5High>3Normal [13:01:00] 6operations, 10MediaWiki-Cache, 6Performance-Team, 10hardware-requests, 7Availability: Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams - https://phabricator.wikimedia.org/T114191#1803397 (10mark) We're not deploying new production services on systems out of warranty. R... [13:01:13] oh no [13:01:21] I have to rebuild all the Zuul .deb packages [13:01:22] :( [13:17:06] 6operations, 10Analytics, 10Traffic: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1803407 (10BBlack) 3NEW [13:18:51] (03PS4) 10BBlack: add X-Client-IP to response headers for vk to consume [puppet] - 10https://gerrit.wikimedia.org/r/252427 (https://phabricator.wikimedia.org/T118557) [13:18:54] (03PS1) 10BBlack: Add resp.http.X-Client-IP -> webrequest:client_ip [puppet] - 10https://gerrit.wikimedia.org/r/252928 (https://phabricator.wikimedia.org/T118557) [13:18:55] (03PS1) 10BBlack: Remove webrequest "ip" and "x_forwarded_for" [puppet] - 10https://gerrit.wikimedia.org/r/252929 (https://phabricator.wikimedia.org/T118557) [13:19:47] (03CR) 10BBlack: "This patch reworked to just add the new response header, followup patches deal with the VK side of things. See bug T118557 if we want to " [puppet] - 10https://gerrit.wikimedia.org/r/252427 (https://phabricator.wikimedia.org/T118557) (owner: 10BBlack) [13:21:49] (03PS5) 10BBlack: add X-Client-IP to response headers for vk to consume [puppet] - 10https://gerrit.wikimedia.org/r/252427 (https://phabricator.wikimedia.org/T118557) [13:21:51] (03PS2) 10BBlack: Add resp.http.X-Client-IP -> webrequest:client_ip [puppet] - 10https://gerrit.wikimedia.org/r/252928 (https://phabricator.wikimedia.org/T118557) [13:21:53] (03PS2) 10BBlack: Remove webrequest "ip" and "x_forwarded_for" [puppet] - 10https://gerrit.wikimedia.org/r/252929 (https://phabricator.wikimedia.org/T118557) [13:30:40] 6operations, 5Continuous-Integration-Scaling: Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie - https://phabricator.wikimedia.org/T118340#1803425 (10hashar) [13:35:26] (03PS1) 10Alexandros Kosiaris: Introduce dubnium.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/252930 (https://phabricator.wikimedia.org/T117183) [13:41:15] addshore: ping [13:41:20] 6operations, 10Analytics, 10Traffic, 5Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1803453 (10BBlack) [13:42:10] (03PS4) 10BBlack: wgHTCPRouting: use separate address for upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/249121 (https://phabricator.wikimedia.org/T116752) [13:46:22] paravoid: pong [13:46:33] in a meeting right now, should be done in the next 15 mins [13:46:55] addshore: can you redirect output of your cronjobs to /dev/null or a logfile of yours or something? [13:46:58] addshore: talking about stat1002 [13:47:14] addshore: I've had to edit your cronjobs a number of times, last time I even left a big comment in caps :P [13:47:50] (03CR) 10Alexandros Kosiaris: [C: 032] Introduce dubnium.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/252930 (https://phabricator.wikimedia.org/T117183) (owner: 10Alexandros Kosiaris) [13:48:26] paravoid: will do now! :/ [13:49:45] thanks :) [13:50:20] done! [13:50:42] I guess my email account doesnt get my cron spam by default then? [13:51:29] all of the cron spam goes to roots [13:51:34] yay! ;) [13:51:42] you can change it with MAILTO= too [13:51:57] * addshore doesnt want to cronspam either ;) [13:52:17] also, the reason I probably missed previous comments is I paste over the cron each time ;) [13:52:38] no worries [13:52:45] irc works too! [13:53:00] :D [13:53:52] (03CR) 10Zhuyifei1999: "> Apart from that, i am wondering why not all url are whitelisted." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252908 (https://phabricator.wikimedia.org/T118554) (owner: 10Zhuyifei1999) [13:54:43] (03PS5) 10BBlack: wgHTCPRouting: use separate address for upload [mediawiki-config] - 10https://gerrit.wikimedia.org/r/249121 (https://phabricator.wikimedia.org/T116752) [13:57:24] (03PS1) 10Alexandros Kosiaris: Introduce dubnium.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/252935 (https://bugzilla.wikimedia.org/117183) [13:59:37] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 9 below the confidence bounds [14:04:47] 6operations, 5Continuous-Integration-Scaling: Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie - https://phabricator.wikimedia.org/T118340#1803515 (10hashar) I had to bump the patch `0005-Cloner-Implement-cache-no-hardlinks-argument.patch` which was outdated. That is for T97106. I h... [14:05:00] 6operations, 5Continuous-Integration-Scaling: Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie - https://phabricator.wikimedia.org/T118340#1803518 (10hashar) [14:12:42] (03PS2) 10Alexandros Kosiaris: Introduce dubnium.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/252935 (https://bugzilla.wikimedia.org/117183) [14:18:32] (03PS6) 10Addshore: Retain daily.* graphite metrics for longer (25y) [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) [14:19:46] (03CR) 10Addshore: "After many discussions in many places we have decided to try and push forward with this config change and start storing data in this graph" [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) (owner: 10Addshore) [14:21:14] I've manually synced the ubuntu repo- this is the second time today that it fails [14:24:47] godog_: ^^ [14:25:03] 6operations, 10Traffic: update the multicast purging documentation - https://phabricator.wikimedia.org/T82096#1803562 (10hashar) [14:27:35] (03CR) 10Alexandros Kosiaris: [C: 032] Introduce dubnium.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/252935 (https://bugzilla.wikimedia.org/117183) (owner: 10Alexandros Kosiaris) [14:29:27] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [14:34:38] 6operations, 10Traffic: update the multicast purging documentation - https://phabricator.wikimedia.org/T82096#1803598 (10hashar) [14:43:30] !log restarting db2059, db2066 for regular mysql upgrade [14:43:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:45:24] addshore: looking [14:48:01] we just had a meeting and right now think thats the best way forward [14:48:31] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, seems the path of least resistance going forward" [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) (owner: 10Addshore) [14:48:44] addshore: I agree, the simplest for now [14:50:55] (03CR) 10Filippo Giunchedi: "also, to be merged on monday" [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) (owner: 10Addshore) [15:03:39] !log restarting db2058, db2065 for regular mysql upgrade [15:03:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:13:17] (03PS1) 10Filippo Giunchedi: cassandra add restbase2001-b cassandra instance [puppet] - 10https://gerrit.wikimedia.org/r/252940 [15:13:36] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [15:14:53] (03PS2) 10Filippo Giunchedi: cassandra: add restbase2001-b instance [puppet] - 10https://gerrit.wikimedia.org/r/252940 [15:15:05] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] cassandra: add restbase2001-b instance [puppet] - 10https://gerrit.wikimedia.org/r/252940 (owner: 10Filippo Giunchedi) [15:16:56] !log bootstrap restbase2001-b cassandra instance [15:17:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:23:38] 6operations, 10Wikimedia-Mailing-lists: Upgrade Mailman to version 3 - https://phabricator.wikimedia.org/T52864#1803748 (10cscott) From @legoktm on IRC: Fedora is switching to Mailman 3 next week: https://fedoraproject.org/wiki/Mailman3_Migration That will satisfy the "actually to be used elsewhere" blocker.... [15:26:59] (03PS1) 10Giuseppe Lavagetto: Revert "Remove nuria and bmansurov's SSH keys" [puppet] - 10https://gerrit.wikimedia.org/r/252941 [15:27:08] (03PS2) 10Giuseppe Lavagetto: Revert "Remove nuria and bmansurov's SSH keys" [puppet] - 10https://gerrit.wikimedia.org/r/252941 [15:27:30] <_joe_> akosiaris: ^^ [15:28:19] <_joe_> anyone has anything against it? [15:28:59] not me [15:29:07] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [15:29:21] seems fine to me (ori moved to a new yubikey-based key since that is planned for opsen anyway) [15:29:54] (03CR) 10Rush: [C: 031] "seems reasonable to me" [puppet] - 10https://gerrit.wikimedia.org/r/252916 (owner: 10Muehlenhoff) [15:30:07] <_joe_> ok merging it then [15:30:31] (03CR) 10Giuseppe Lavagetto: [C: 032] Revert "Remove nuria and bmansurov's SSH keys" [puppet] - 10https://gerrit.wikimedia.org/r/252941 (owner: 10Giuseppe Lavagetto) [15:31:37] _joe_: not me [15:32:21] 6operations, 10MediaWiki-Cache, 6Performance-Team, 10hardware-requests, 7Availability: Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams - https://phabricator.wikimedia.org/T114191#1803766 (10GWicke) @mark, @robh: What is the timeline for this procurement? Are those serve... [15:32:33] <_joe_> ok good! [15:33:46] 6operations, 10MediaWiki-Cache, 6Performance-Team, 10hardware-requests, 7Availability: Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams - https://phabricator.wikimedia.org/T114191#1803770 (10RobH) a:5mark>3RobH I'm currently working on the misc system quotes, I expec... [15:34:06] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [15:35:47] PROBLEM - cassandra-b CQL 10.192.16.163:9042 on restbase2001 is CRITICAL: Connection refused [15:35:57] that's me ^ known [15:36:46] ACKNOWLEDGEMENT - cassandra-b CQL 10.192.16.163:9042 on restbase2001 is CRITICAL: Connection refused Filippo Giunchedi instance bootstrapping [15:36:54] 6operations, 10MediaWiki-Cache, 6Performance-Team, 10hardware-requests, 7Availability: Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams - https://phabricator.wikimedia.org/T114191#1803777 (10faidon) >>! In T114191#1778608, @Ottomata wrote: > Mmk! > > @GWicke, I can't re... [15:37:27] PROBLEM - Corp OIT LDAP Mirror on dubnium is CRITICAL: Could not bind to the LDAP server [15:38:07] ^ ignore [15:38:18] that's used for exim? [15:38:29] mark: no, not yet [15:38:37] it's https://phabricator.wikimedia.org/T117182 [15:38:38] sigh [15:38:40] my fault [15:38:52] page is a false alarm, sorry people [15:39:06] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [15:39:10] ack [15:39:15] interface::add_ip6_mapped - can this not be used in labs? [15:40:17] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [15:40:19] i hear it breaks things in beta cluster [15:41:17] PROBLEM - puppet last run on dubnium is CRITICAL: CRITICAL: Puppet has 1 failures [15:44:06] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [15:44:45] 6operations, 7Graphite: http 500 errors from check_graphite on rate of media upload checks - https://phabricator.wikimedia.org/T118398#1803786 (10fgiunchedi) I stand corrected, it seems icinga is silently stripping single quotes when exec-ing the command ``` [pid 12449] execve("/usr/lib/nagios/plugins/check_g... [15:47:07] RECOVERY - puppet last run on dubnium is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [15:49:06] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 110 seconds ago with 0 failures [15:50:23] ok dubnium fixed [15:50:27] (03PS1) 10Faidon Liambotis: Add gw PTR for frack-payments-{eqiad,codfw} [dns] - 10https://gerrit.wikimedia.org/r/252944 [15:50:29] (03PS1) 10Faidon Liambotis: Add forward DNS for payments-codfw.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/252945 [15:51:18] (03PS2) 10Faidon Liambotis: Add forward DNS for payments-codfw.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/252945 [15:51:20] (03PS2) 10Faidon Liambotis: Add gw PTR for frack-payments-{eqiad,codfw} [dns] - 10https://gerrit.wikimedia.org/r/252944 [15:53:41] 6operations, 5Patch-For-Review: migrate pollux/plutonium into VMs - https://phabricator.wikimedia.org/T117182#1768226 (10akosiaris) [15:53:42] 6operations, 10vm-requests, 5Patch-For-Review: Site: 2 VM request for OIT LDAP mirror - https://phabricator.wikimedia.org/T117183#1803791 (10akosiaris) 5Open>3Resolved a:3akosiaris dubnium was installed today as a replacement for plutonium. Closing this task, the actual migration is on T117182 [15:53:50] (03CR) 10Faidon Liambotis: [C: 032] Add gw PTR for frack-payments-{eqiad,codfw} [dns] - 10https://gerrit.wikimedia.org/r/252944 (owner: 10Faidon Liambotis) [15:54:41] (03CR) 10Faidon Liambotis: [C: 032] Add forward DNS for payments-codfw.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/252945 (owner: 10Faidon Liambotis) [15:55:00] (03PS1) 10Alexandros Kosiaris: ldap-corp.eqiad: Point to dubnium instead [dns] - 10https://gerrit.wikimedia.org/r/252946 [15:55:03] mutante: fwiw, link-local ipv6 addresses worked when I tested them a while ago, so at least openstack is able to route ipv6 packets [15:56:13] valhallasw`cloud: hmm.. thanks. though there is the claim that "Breaks deployment-bastion puppet in beta cluster." [15:56:46] Maybe openstack doesn't understand fixed-configured ipv6 addresses? Not sure. Might also be a version issue [15:56:55] http://docs.openstack.org/networking-guide/adv_config_ipv6.html has a lot of information, but I don't get most of it [15:56:59] openstack doesn't exactly support IPv6 [15:57:06] * valhallasw`cloud should really read an ipv6 primer at some point [15:57:06] it does, in some configuration since juno [15:57:44] my take is assume IPv6 is broken currently in openstack [15:57:46] yeah, according to that page they added stuff in kilo [15:57:49] unless you feel like debugging it [15:58:05] and find out what is only half baked in [15:58:08] *nod* [16:00:26] (03PS2) 10Dzahn: Revert "deployment::server: move IPv6 int to role" [puppet] - 10https://gerrit.wikimedia.org/r/252732 (https://phabricator.wikimedia.org/T118422) (owner: 10Thcipriani) [16:00:26] hmmm. ok. .. then ... [16:01:02] (03CR) 10Dzahn: [C: 032] Revert "deployment::server: move IPv6 int to role" [puppet] - 10https://gerrit.wikimedia.org/r/252732 (https://phabricator.wikimedia.org/T118422) (owner: 10Thcipriani) [16:01:03] akosiaris: is that dubnium setup already live? we'll also need to fix the ferm rule (which referenced plutonium.wikimedia.org instead of querying the copr-LDAP servers from Hiera) [16:01:11] 6operations, 5Patch-For-Review: migrate pollux/plutonium into VMs - https://phabricator.wikimedia.org/T117182#1803801 (10akosiaris) Instead of migrating plutonium as is, dubnium was introduced to the cluster today. It is already ready to accept connections. I will configure exim to use this VM instead of pluto... [16:01:38] (03PS1) 10Faidon Liambotis: Fix payments.{eqiad,codfw} to be payments-{eqiad,codfw} [dns] - 10https://gerrit.wikimedia.org/r/252949 [16:01:52] (03PS2) 10Alexandros Kosiaris: ldap-corp.eqiad: Point to dubnium instead [dns] - 10https://gerrit.wikimedia.org/r/252946 (https://phabricator.wikimedia.org/T117182) [16:01:55] 6operations, 10Analytics, 10Traffic, 5Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1803813 (10Ottomata) > Add client_ip field based on the above to the webrequest varnishkafka log format (We can probably start getting this into e.g. oxy... [16:01:58] (03CR) 10Faidon Liambotis: [C: 032] Fix payments.{eqiad,codfw} to be payments-{eqiad,codfw} [dns] - 10https://gerrit.wikimedia.org/r/252949 (owner: 10Faidon Liambotis) [16:02:13] moritzm: live: yes, used: no [16:02:14] (03PS3) 10Faidon Liambotis: ldap-corp.eqiad: Point to dubnium instead [dns] - 10https://gerrit.wikimedia.org/r/252946 (https://phabricator.wikimedia.org/T117182) (owner: 10Alexandros Kosiaris) [16:02:25] moritzm: good catch, I 'll amend the ferm rules [16:02:37] akosiaris: shall I merge? [16:02:46] also, you keep forgetting IPv6 for these things :) [16:03:12] I don't, it's on purpose. I 'll add it later on [16:03:37] paravoid: yeah merge [16:03:43] thanks [16:04:02] (03CR) 10Faidon Liambotis: [C: 032] ldap-corp.eqiad: Point to dubnium instead [dns] - 10https://gerrit.wikimedia.org/r/252946 (https://phabricator.wikimedia.org/T117182) (owner: 10Alexandros Kosiaris) [16:04:21] is https://gerrit.wikimedia.org/r/#/c/250438/ also ready to be merged? [16:04:37] nope [16:06:25] (03PS1) 10Alexandros Kosiaris: role::openldap::corp: Add dubnium.wikimedia.org in ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/252951 [16:06:59] (03PS2) 10Dzahn: zookeeper,dynamicproxy: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252389 [16:07:17] (03CR) 10Alexandros Kosiaris: [C: 032] role::openldap::corp: Add dubnium.wikimedia.org in ferm rules [puppet] - 10https://gerrit.wikimedia.org/r/252951 (owner: 10Alexandros Kosiaris) [16:08:37] (03CR) 10Dzahn: [C: 032] zookeeper,dynamicproxy: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252389 (owner: 10Dzahn) [16:08:46] (03PS3) 10Dzahn: zookeeper,dynamicproxy: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252389 [16:10:52] paravoid: I am gonna do a switchover manually to ldap-corp.$::site, first on mx2001 and if all goes well on mx1001 [16:11:38] sounds a bit paranoid, but whatever works for you :) [16:11:59] I am a bit paranoid, but kind of afraid of breaking everyone's email for a long time [16:13:58] <_joe_> akosiaris: not getting emails in general could be bad, can we do that on a per-list basis? [16:14:12] <_joe_> (breaking email for a long time) [16:14:54] (03Abandoned) 10EBernhardson: Sample CirrusSearchRequestSet at 1:10000 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252879 (owner: 10EBernhardson) [16:16:21] 6operations, 10EventBus, 10MediaWiki-Cache, 6Performance-Team, and 2 others: Setup a 2 server Kafka instance in both eqiad and codfw for reliable purge streams - https://phabricator.wikimedia.org/T114191#1803833 (10Ottomata) [16:16:38] !log restarting mysql on db2056, db2063 and db2064 for regular upgrade [16:16:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:19:33] _joe_: nope [16:19:41] either I break everything or I dont [16:23:51] <_joe_> akosiaris: too bad [16:23:56] (03CR) 10Hashar: "Merged upstream https://review.openstack.org/#/c/222689/" [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/237699 (owner: 10Hashar) [16:27:33] (03PS2) 10Giuseppe Lavagetto: base::certificates: add puppet's CA to the trusted store [puppet] - 10https://gerrit.wikimedia.org/r/252681 (https://phabricator.wikimedia.org/T114638) [16:31:48] (03PS1) 10Hashar: node deletion delay is now configurable [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/252953 [16:32:38] (03CR) 10Hashar: [C: 04-2] "Cherry pick of https://review.openstack.org/#/c/245220/ for T113359." [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/252953 (owner: 10Hashar) [16:32:55] 6operations, 10ops-eqiad: Decommission sodium - https://phabricator.wikimedia.org/T110142#1803868 (10Cmjohnson) 5Open>3Resolved Wiped and removed from the rack [16:33:26] 6operations, 10ops-eqiad: check for spare 4GB DDR3 Synchronous 1333 MHz RAM - https://phabricator.wikimedia.org/T117248#1803870 (10Cmjohnson) 5Open>3Resolved [16:33:27] 6operations, 6Parsing-Team, 10hardware-requests: Dedicated server for running Parsoid's roundtrip tests to get reliable parse latencies and use as perf. benchmarking tests - https://phabricator.wikimedia.org/T116090#1803871 (10Cmjohnson) [16:33:47] (03PS2) 10Hashar: Continue image refresh if /etc/nodepool exists [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/237699 [16:33:49] (03PS2) 10Hashar: Add 'debian' user in bootstrapServer() [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/237381 [16:33:51] (03PS2) 10Hashar: Convert to use latest statsd version [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/235691 [16:33:53] (03PS2) 10Hashar: Debug dying task managers [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/235690 [16:33:55] (03PS2) 10Hashar: Convert timing metrics to milliseconds [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/235692 [16:33:57] (03PS2) 10Hashar: node deletion delay is now configurable [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/252953 [16:33:59] (03PS8) 10Hashar: Stop all threads on SIGUSR1 [debs/nodepool] (patch-queue/debian) - 10https://gerrit.wikimedia.org/r/225410 [16:35:56] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [500.0] [16:42:05] the alert above seems real, nice 5xx spike [16:42:24] 503 specifically [16:43:07] all esams + text cluster heh [16:43:22] so maybe a network hiccup of some kind [16:46:49] (03PS2) 10Hashar: nodepool 0.1.1-wmf4 [debs/nodepool] (debian) - 10https://gerrit.wikimedia.org/r/237700 (https://phabricator.wikimedia.org/T111377) [16:53:14] (03CR) 10Hashar: "Build and pushed at https://people.wikimedia.org/~hashar/debs/nodepool_0.1.1-wmf4/" [debs/nodepool] (debian) - 10https://gerrit.wikimedia.org/r/237700 (https://phabricator.wikimedia.org/T111377) (owner: 10Hashar) [16:53:34] (03CR) 10JanZerebecki: [C: 031] Retain daily.* graphite metrics for longer (25y) [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) (owner: 10Addshore) [16:56:02] 6operations, 10ops-eqiad: Decommission cisco servers, Analytics1003, 1004 and 1010 - https://phabricator.wikimedia.org/T118572#1803903 (10Cmjohnson) 3NEW [16:56:37] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [16:57:41] (03PS1) 10Filippo Giunchedi: swift: use double quote in graphite query [puppet] - 10https://gerrit.wikimedia.org/r/252955 (https://phabricator.wikimedia.org/T118398) [16:58:04] bblack: we don't see similar spikes for simple internal API requests [16:58:45] 6operations, 10ops-eqiad: Decommission cisco servers, Analytics1003, 1004 and 1010 - https://phabricator.wikimedia.org/T118572#1803914 (10Ottomata) NOooOOOooooO.....Okay. +1 :( :) [16:58:53] there was another at 11:30 UTC [16:59:14] gwicke: yeah the spike was only for esams text requests, not the other DCs, so likely the 503s were from a network hiccup for esams<->eqiad [16:59:26] 6operations, 10ops-eqiad: Decommission cisco servers, Analytics1003, 1004 and 1010 - https://phabricator.wikimedia.org/T118572#1803915 (10Ottomata) [16:59:37] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: use double quote in graphite query [puppet] - 10https://gerrit.wikimedia.org/r/252955 (https://phabricator.wikimedia.org/T118398) (owner: 10Filippo Giunchedi) [17:00:05] bblack: "the network is reliable" [17:00:42] https://grafana.wikimedia.org/dashboard/db/varnish-aggregate-client-status-codes?from=1447412436053&to=1447434036053&var-site=esams&var-cache_type=text&var-status_type=5 [17:00:50] ^ that shows the 11:30 and the more-recent ones [17:01:02] if you explore around the selectors at the top, it seems isolated to esams+text [17:01:47] those multi-selectors are neat [17:02:47] 6operations, 5Patch-For-Review, 7Swift: install/setup/deploy ms-be2016-2021 - https://phabricator.wikimedia.org/T116842#1803921 (10fgiunchedi) all machines in service at weight 3000, pending full provisioning and testing of eqiad at weight 4000 [17:04:35] paravoid: around ? [17:04:42] yes [17:04:53] I think I just stumbled across a gnu-tls bug or something [17:05:07] need a second pair of eyes [17:05:18] shoot [17:06:34] so ldapsearch -x -ZZ -h ldap-corp.eqiad.wikimedia.org -b "dc=corp,dc=wikimedia,dc=org" uid=akosiaris on plutonium(14.04 ubuntu). works fine. same command on mx1001, mx1002, dubnium or pollux fails with: TLS: peer cert untrusted or revoked (0x102) [17:06:35] 6operations, 5Continuous-Integration-Scaling, 5Patch-For-Review: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1803924 (10hashar) I think most of the work has been done now. I have poked by email @andrew and @chasemp to figure out with whom / when I can handle... [17:06:57] same command with argument ldap-mirror.wikimedia.org works fine everywhere [17:07:45] maybe not a gnutls bug. I see 14.04's ldapsearch is linked against gnutls as well [17:08:15] no 636? [17:08:25] startTLS [17:08:29] cert is fine [17:08:34] I 've looked it multiple times [17:08:45] but please look at it again [17:08:50] I might be blind or something [17:09:06] ldapsearch -d2 if you want packet traces [17:09:21] or -d255 -v if you want full debugging [17:09:26] can we do open up 636? [17:09:34] s/do// [17:09:36] temporarily ? yes [17:09:39] yes please [17:11:03] done [17:11:44] faidon@mx1001:~$ openssl s_client -connect ldap-corp.eqiad.wikimedia.org:636 [17:11:47] CONNECTED(00000003) [17:11:49] 140673663792784:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: [17:11:52] --- [17:11:55] no peer certificate available [17:13:37] not sure how this makes any sense... [17:13:44] (03PS1) 10DCausse: Add 2 payloads map fields to CirrusSearchRequestSet avro schema [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252957 (https://phabricator.wikimedia.org/T118570) [17:13:57] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 30.77% of data above the critical threshold [500.0] [17:15:37] conn=1006 fd=15 closed (TLS negotiation failure) [17:15:44] yeah [17:15:44] sec [17:17:07] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [17:19:06] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [17:24:35] (03CR) 10EBernhardson: [C: 031] "To deploy without making the serializers complain I think we need to deploy the new field starting at the end of the pipeline (camus) whic" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252957 (https://phabricator.wikimedia.org/T118570) (owner: 10DCausse) [17:26:59] gnutls-cli says [17:27:03] paravoid: so gnutls-bin says "he certificate chain uses insecure algorithm." [17:27:05] - Status: The certificate is NOT trusted. The certificate chain uses insecure algorithm. [17:27:08] sigh [17:27:08] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [17:27:08] heh [17:27:08] lol [17:27:30] oh damn... [17:27:50] I made sure to sign ldap-mirror.eqiad with sha256, but I forgot it on ldap-corp.eqiad [17:27:52] pfff [17:27:53] ok [17:27:56] paravoid: thanks!!! [17:29:10] paravoid: see now why I was paranoid ? [17:29:30] ldap-corp.eqiad is sha256 too [17:30:08] but [17:30:10] oh, it's the root ? [17:30:11] ldap-corp.eqiad is... [17:30:20] a 512-bit RSA certificate :) [17:30:23] lol [17:30:30] so is ldap-corp-codfw [17:30:44] pfff ok [17:30:46] fixing [17:30:54] so I'm so glad gnutls doesn't accept it [17:37:49] !log applied and reverted patch for T118032 (lots of exceptions on deploy) [17:37:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:38:57] 6operations, 6Analytics-Backlog, 6Analytics-Kanban, 7Monitoring, 5Patch-For-Review: Turn off sqstat udp2log instance - https://phabricator.wikimedia.org/T117727#1804028 (10Ottomata) [17:39:00] 6operations, 10Analytics-Cluster, 5Patch-For-Review: Turn off webrequest udp2log instances. - https://phabricator.wikimedia.org/T97294#1804027 (10Ottomata) [17:39:38] 6operations, 10Analytics-Cluster, 6Analytics-Kanban, 5Patch-For-Review: Turn off webrequest udp2log instances. - https://phabricator.wikimedia.org/T97294#1804032 (10Ottomata) p:5Triage>3Normal [17:40:10] 6operations, 10Analytics-Cluster, 6Analytics-Kanban, 5Patch-For-Review: Turn off webrequest udp2log instances. - https://phabricator.wikimedia.org/T97294#1238209 (10Ottomata) Let's turn of the udp2log instance on erbium. @Jeff_Green, if you say this ok, we will do it! [17:40:11] (03PS1) 10Faidon Liambotis: Remove role::logging::udp2log::erbium and friends [puppet] - 10https://gerrit.wikimedia.org/r/252961 [17:40:30] (03PS2) 10Faidon Liambotis: Remove role::logging::udp2log::erbium and friends [puppet] - 10https://gerrit.wikimedia.org/r/252961 (https://phabricator.wikimedia.org/T97294) [17:41:04] (03CR) 10jenkins-bot: [V: 04-1] Remove role::logging::udp2log::erbium and friends [puppet] - 10https://gerrit.wikimedia.org/r/252961 (https://phabricator.wikimedia.org/T97294) (owner: 10Faidon Liambotis) [17:41:07] (03PS3) 10Ottomata: Remove role::logging::udp2log::erbium and friends [puppet] - 10https://gerrit.wikimedia.org/r/252961 (https://phabricator.wikimedia.org/T84062) (owner: 10Faidon Liambotis) [17:41:12] pfft [17:41:16] 6operations, 10Analytics-Cluster, 6Analytics-Kanban, 5Patch-For-Review: Turn off webrequest udp2log instances. - https://phabricator.wikimedia.org/T97294#1804040 (10Jgreen) >>! In T97294#1804032, @Ottomata wrote: > Let's turn of the udp2log instance on erbium. @Jeff_Green, if you say this ok, we will do i... [17:42:01] (03CR) 10jenkins-bot: [V: 04-1] Remove role::logging::udp2log::erbium and friends [puppet] - 10https://gerrit.wikimedia.org/r/252961 (https://phabricator.wikimedia.org/T84062) (owner: 10Faidon Liambotis) [17:43:06] (03PS4) 10Faidon Liambotis: Remove role::logging::udp2log::erbium and friends [puppet] - 10https://gerrit.wikimedia.org/r/252961 (https://phabricator.wikimedia.org/T84062) [17:49:23] (03PS1) 10Alexandros Kosiaris: ldap-corp: Update to the new re-issued certificates [puppet] - 10https://gerrit.wikimedia.org/r/252962 [17:50:35] (03CR) 10Alexandros Kosiaris: [C: 032] ldap-corp: Update to the new re-issued certificates [puppet] - 10https://gerrit.wikimedia.org/r/252962 (owner: 10Alexandros Kosiaris) [17:55:10] (03PS1) 10Filippo Giunchedi: monitoring: fail on graphite metrics using single quotes [puppet] - 10https://gerrit.wikimedia.org/r/252963 (https://phabricator.wikimedia.org/T118398) [17:55:19] paravoid: ok new keys worked fine [17:55:30] proceeding with the plan [17:56:00] (03CR) 10jenkins-bot: [V: 04-1] monitoring: fail on graphite metrics using single quotes [puppet] - 10https://gerrit.wikimedia.org/r/252963 (https://phabricator.wikimedia.org/T118398) (owner: 10Filippo Giunchedi) [17:56:25] 6operations, 5Patch-For-Review: Add monitoring of upload rate on commons to icinga alerts - https://phabricator.wikimedia.org/T92322#1804096 (10fgiunchedi) 5Open>3Resolved fixed with https://gerrit.wikimedia.org/r/252955 [17:56:50] (03CR) 10Dereckson: [C: 04-1] "-1 per CSteipp comment in Phabricator." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252908 (https://phabricator.wikimedia.org/T118554) (owner: 10Zhuyifei1999) [17:58:00] (03PS2) 10Filippo Giunchedi: monitoring: fail on graphite metrics using single quotes [puppet] - 10https://gerrit.wikimedia.org/r/252963 (https://phabricator.wikimedia.org/T118398) [18:03:27] (03CR) 10Alexandros Kosiaris: [C: 032] exim: removal of non-DC aware ldap-mirror CNAME [puppet] - 10https://gerrit.wikimedia.org/r/250438 (owner: 10Alexandros Kosiaris) [18:03:41] (03PS3) 10Alexandros Kosiaris: exim: removal of non-DC aware ldap-mirror CNAME [puppet] - 10https://gerrit.wikimedia.org/r/250438 [18:03:52] (03PS4) 10Alexandros Kosiaris: exim: removal of non-DC aware ldap-mirror CNAME [puppet] - 10https://gerrit.wikimedia.org/r/250438 [18:04:34] (03CR) 10jenkins-bot: [V: 04-1] exim: removal of non-DC aware ldap-mirror CNAME [puppet] - 10https://gerrit.wikimedia.org/r/250438 (owner: 10Alexandros Kosiaris) [18:04:50] !log added neodymium to salt term in common-infrastructure4 on cr[12]-eqiad [18:04:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:05:05] ^ in case that de-syncs your router config scripts or whatever [18:05:59] (03CR) 10Alexandros Kosiaris: [V: 032] exim: removal of non-DC aware ldap-mirror CNAME [puppet] - 10https://gerrit.wikimedia.org/r/250438 (owner: 10Alexandros Kosiaris) [18:06:23] that jenkins bot -1 was bogus ... [18:09:19] 6operations, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10MediaWiki-extensions-Translate: Publishing translations for central notice banners fails - https://phabricator.wikimedia.org/T104774#1804186 (10awight) [18:10:56] paravoid: dubnium seems to be holding ok [18:11:17] that migration's difficult part is done, now off to getting a broom and cleaning up [18:13:43] 6operations, 5Patch-For-Review: Set up a ou=corp LDAP replica for redundancy - https://phabricator.wikimedia.org/T82662#1804217 (10akosiaris) 5Open>3Resolved And with https://gerrit.wikimedia.org/r/#/c/250438/ merged, the migration to a redundant LDAP schema is now complete. Resolving [18:17:06] (03PS1) 10Ottomata: Initial deb packaging [debs/python-dotted] (debian) - 10https://gerrit.wikimedia.org/r/252967 [18:17:29] (03PS2) 10Ottomata: Initial deb packaging [debs/python-dotted] (debian) - 10https://gerrit.wikimedia.org/r/252967 (https://phabricator.wikimedia.org/T118578) [18:19:41] 6operations, 10ops-eqiad: reclaim plutonium to spares - https://phabricator.wikimedia.org/T118586#1804229 (10akosiaris) 3NEW [18:20:54] 6operations, 10ops-eqiad: reclaim plutonium to spares - https://phabricator.wikimedia.org/T118586#1804244 (10akosiaris) [18:20:55] (03Abandoned) 10Ori.livneh: redis: add doc-block [puppet] - 10https://gerrit.wikimedia.org/r/252646 (owner: 10Ori.livneh) [18:23:22] (03PS3) 10Alexandros Kosiaris: ldap-mirror: Remove the old non-DC specific CNAME [dns] - 10https://gerrit.wikimedia.org/r/249829 [18:23:27] (03PS4) 10Alexandros Kosiaris: ldap-mirror: Remove the old non-DC specific CNAME [dns] - 10https://gerrit.wikimedia.org/r/249829 [18:23:39] (03CR) 10jenkins-bot: [V: 04-1] ldap-mirror: Remove the old non-DC specific CNAME [dns] - 10https://gerrit.wikimedia.org/r/249829 (owner: 10Alexandros Kosiaris) [18:27:19] (03CR) 10Alexandros Kosiaris: [C: 032] ldap-mirror: Remove the old non-DC specific CNAME [dns] - 10https://gerrit.wikimedia.org/r/249829 (owner: 10Alexandros Kosiaris) [18:27:51] (03PS2) 10Alexandros Kosiaris: ldap-mirror: Remove the old DC specific names [dns] - 10https://gerrit.wikimedia.org/r/250415 [18:28:09] (03CR) 10jenkins-bot: [V: 04-1] ldap-mirror: Remove the old DC specific names [dns] - 10https://gerrit.wikimedia.org/r/250415 (owner: 10Alexandros Kosiaris) [18:30:49] 6operations: Are production boxes in codfw in realm 'production? - https://phabricator.wikimedia.org/T118590#1804277 (10Andrew) 3NEW [18:32:41] 6operations: Are production boxes in codfw in realm 'production? - https://phabricator.wikimedia.org/T118590#1804286 (10Andrew) 5Open>3Resolved a:3Andrew nevermind! [18:34:26] 6operations, 10fundraising-tech-ops, 5Patch-For-Review: remove fundraising banner log related cruft from production puppet - https://phabricator.wikimedia.org/T118325#1804300 (10Jgreen) [18:34:29] 6operations, 10Analytics-Cluster, 6Analytics-Kanban, 5Patch-For-Review: Turn off webrequest udp2log instances. - https://phabricator.wikimedia.org/T97294#1804301 (10Jgreen) [18:34:31] 6operations, 10Fundraising-Backlog, 6Security, 10fundraising-tech-ops: Delete gadolinium:/a/log/fundraising/ - https://phabricator.wikimedia.org/T92336#1804302 (10Jgreen) [18:34:40] (03PS4) 10Alexandros Kosiaris: ldap-mirror: Remove the vary on DC name to complete the migration [puppet] - 10https://gerrit.wikimedia.org/r/250420 [18:34:42] (03PS1) 10Alexandros Kosiaris: Remove plutonium from infrastructure [puppet] - 10https://gerrit.wikimedia.org/r/252968 (https://phabricator.wikimedia.org/T118586) [18:37:19] (03CR) 10Alexandros Kosiaris: [C: 032] ldap-mirror: Remove the vary on DC name to complete the migration [puppet] - 10https://gerrit.wikimedia.org/r/250420 (owner: 10Alexandros Kosiaris) [18:38:11] PROBLEM - puppet last run on dubnium is CRITICAL: CRITICAL: Puppet has 1 failures [18:43:03] (03PS1) 10Alexandros Kosiaris: role::openldap::corp: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/252969 [18:43:54] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] role::openldap::corp: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/252969 (owner: 10Alexandros Kosiaris) [18:47:22] RECOVERY - puppet last run on dubnium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:07:49] (03PS3) 10Alexandros Kosiaris: ldap-mirror: Remove the old DC specific names [dns] - 10https://gerrit.wikimedia.org/r/250415 [19:09:59] (03PS4) 10Alexandros Kosiaris: ldap-mirror: Remove the old DC specific names [dns] - 10https://gerrit.wikimedia.org/r/250415 [19:11:44] (03CR) 10Alexandros Kosiaris: [C: 032] ldap-mirror: Remove the old DC specific names [dns] - 10https://gerrit.wikimedia.org/r/250415 (owner: 10Alexandros Kosiaris) [19:27:32] (03PS1) 10Alexandros Kosiaris: Reclaim plutonium.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/252972 (https://bugzilla.wikimedia.org/118586) [19:28:37] 6operations, 10ops-eqiad, 5Patch-For-Review: reclaim plutonium to spares - https://phabricator.wikimedia.org/T118586#1804428 (10akosiaris) [19:35:52] 6operations, 5Patch-For-Review: migrate pollux/plutonium into VMs - https://phabricator.wikimedia.org/T117182#1804457 (10akosiaris) 5Open>3Resolved Done. plutonium could not be //migrated// and was instead //replaced// by dubnium.wikimedia.org. plutonium is being reclaimed and added to spares in T118586 [19:39:20] 6operations, 10ops-eqiad: remove patch cable for cr1-eqiad:xe-4/2/1 ID 3482, circuit ID ETYX/084858//ZYO - https://phabricator.wikimedia.org/T118177#1804474 (10Cmjohnson) 5Open>3Resolved Patch has been pulled [19:50:25] (03CR) 10JanZerebecki: [C: 04-1] Retain daily.* graphite metrics for longer (25y) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) (owner: 10Addshore) [19:56:22] (03PS7) 10Addshore: Retain daily.* graphite metrics for longer (25y) [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) [20:05:33] (03CR) 10JanZerebecki: [C: 031] Retain daily.* graphite metrics for longer (25y) [puppet] - 10https://gerrit.wikimedia.org/r/247866 (https://phabricator.wikimedia.org/T117402) (owner: 10Addshore) [20:14:16] !log installing xfp optics into cr1-ulsfo:xe-0/0/2 (spare port) [20:14:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:17:25] !log cr1-ulsfo:xe-0/0/2 (spare port) xfp optic detects fine, now installing other spare into cr2-xe-1/3/0 [20:17:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:18:46] !log cr1-ulsfo:xe-0/0/2 optic install completed (detects fine) [20:18:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:19:00] blah [20:19:35] !log cr2-xe-1/3/0 optic install completed (detects fine) [20:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:21:47] (03CR) 10Ottomata: [C: 032] Initial deb packaging [debs/python-dotted] (debian) - 10https://gerrit.wikimedia.org/r/252967 (https://phabricator.wikimedia.org/T118578) (owner: 10Ottomata) [20:21:53] (03CR) 10Ottomata: [V: 032] Initial deb packaging [debs/python-dotted] (debian) - 10https://gerrit.wikimedia.org/r/252967 (https://phabricator.wikimedia.org/T118578) (owner: 10Ottomata) [20:27:22] (03PS1) 10MaxSem: Bump portals [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252986 [20:28:05] (03CR) 10MaxSem: [C: 032] Bump portals [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252986 (owner: 10MaxSem) [20:28:30] (03Merged) 10jenkins-bot: Bump portals [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252986 (owner: 10MaxSem) [20:30:31] !log maxsem@tin Synchronized portals/: Bump to master (duration: 00m 39s) [20:30:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:32:54] Krenair, umaaaaaaaaaaask! [20:34:30] Krenair, nvm [20:34:31] :P [20:35:32] !log maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s) [20:35:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:37:54] MaxSem, ... what? [20:38:09] okay... [20:38:22] !log maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s) [20:38:24] never mind, I'm trying to figure out why scap it pooping a pile of warnings at me:P [20:38:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:39:48] !log maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 29s) [20:41:43] ebernhardson, could you run `cd /srv/mediawiki-staging/.git/modules/portals && find -type f -exec chmod 664 {} \;` on tin? [20:41:53] some files you created are read-only [20:41:55] :P [20:56:33] PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [5000000.0] [21:14:07] MaxSem: just got back, sure i can do that [21:14:11] PROBLEM - etherpad.wikimedia.org HTTP on etherpad1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:17] MaxSem: got a whole ton of operation not permitted, which leads me to believe the general permission handling here is incorrect. but i'd have to dig into it to figure out what is the right way [21:15:54] MaxSem: but it still might have fixed some of the other ones :) try now [21:16:05] can someone restart etherpad? [21:17:21] RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 1.00% above the threshold [1000000.0] [21:17:22] !log bounced etherpad again [21:17:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:17:30] maybe I should setup monit to autorestart it [21:17:41] RECOVERY - etherpad.wikimedia.org HTTP on etherpad1001 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 522 bytes in 0.014 second response time [21:23:15] !log maxsem@tin Synchronized portals/: Did scap errors go away? (duration: 00m 30s) [21:23:26] ebernhardson, meh - still a pile of errors [21:23:41] whatever, shit still gets pushed:P [21:28:57] MaxSem: errors from the sync to mira are expected right now -- https://phabricator.wikimedia.org/T117016 [21:31:09] bd808, my problem looks different: https://gist.github.com/MaxSem/4547de5ed10de99baa37 [21:33:43] It's related. sync-masters is the rsync fetch from tin to mira and it is flakey at the moment. We had it working but it took root help to twiddle permissions and that breaks as soon as there is new git activity on tin. Bad design from me :/ [21:34:03] the tl;dr is that failures of sync-masters are noise right now [21:36:27] bd808: oh jynus (dba) poked me about it [21:39:08] (03PS1) 10Jhobs: Disable first QuickSurveys survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253038 (https://phabricator.wikimedia.org/T118525) [21:39:12] ah https://phabricator.wikimedia.org/T118555 Mira failed to sync [21:40:49] made a dupe [21:40:58] thanks [21:43:43] (03PS2) 10Florianschmidtwelzow: Disable first QuickSurveys survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/253038 (https://phabricator.wikimedia.org/T118525) (owner: 10Jhobs) [21:49:56] (03PS1) 10BryanDavis: scap: Create wrapper script for master-master rsync [puppet] - 10https://gerrit.wikimedia.org/r/253040 (https://phabricator.wikimedia.org/T117016) [22:03:47] 6operations, 10Analytics, 10Traffic, 5Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1804749 (10BBlack) >>! In T118557#1803813, @Ottomata wrote: > Hm, not sure about this. `ip`, maybe. 'ip' is mostly-pointless even now, as it's almost... [22:04:23] (03PS1) 10Ori.livneh: redis: on systemd hosts, provision a systemd unit template [puppet] - 10https://gerrit.wikimedia.org/r/253043 [22:12:12] PROBLEM - Kafka Broker Replica Max Lag on kafka1014 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000000.0] [22:17:27] (03PS1) 10Rush: hiera_lookup -h adjust [puppet] - 10https://gerrit.wikimedia.org/r/253044 [22:18:38] (03PS2) 10Rush: hiera_lookup -h adjust [puppet] - 10https://gerrit.wikimedia.org/r/253044 [22:20:17] (03PS3) 10Rush: hiera_lookup -h adjust [puppet] - 10https://gerrit.wikimedia.org/r/253044 [22:20:38] (03CR) 10BryanDavis: "Associated scap change at https://phabricator.wikimedia.org/D48" [puppet] - 10https://gerrit.wikimedia.org/r/253040 (https://phabricator.wikimedia.org/T117016) (owner: 10BryanDavis) [22:21:10] (03PS4) 10Rush: hiera_lookup -h adjust [puppet] - 10https://gerrit.wikimedia.org/r/253044 [22:24:34] (03CR) 10Rush: [C: 032] hiera_lookup -h adjust [puppet] - 10https://gerrit.wikimedia.org/r/253044 (owner: 10Rush) [22:25:39] (03PS2) 10Ori.livneh: redis: on systemd hosts, provision a systemd unit template [puppet] - 10https://gerrit.wikimedia.org/r/253043 [22:25:46] (03CR) 10Ori.livneh: [C: 032 V: 032] redis: on systemd hosts, provision a systemd unit template [puppet] - 10https://gerrit.wikimedia.org/r/253043 (owner: 10Ori.livneh) [22:29:01] RECOVERY - Kafka Broker Replica Max Lag on kafka1014 is OK: OK: Less than 1.00% above the threshold [1000000.0] [22:42:56] (03PS1) 10Yuvipanda: etherpad: Add an autorestarter [puppet] - 10https://gerrit.wikimedia.org/r/253048 [22:46:08] (03PS4) 10Dzahn: zookeeper,dynamicproxy: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252389 [22:46:13] PROBLEM - citoid endpoints health on sca1002 is CRITICAL: /api is CRITICAL: Could not fetch url http://10.64.48.29:1970/api: Timeout on connection while downloading http://10.64.48.29:1970/api [22:48:02] RECOVERY - citoid endpoints health on sca1002 is OK: All endpoints are healthy [22:53:34] (03PS2) 10Dzahn: role/lvs: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252390 [22:54:46] (03CR) 10Dzahn: [C: 032] "only touches monitoring groups" [puppet] - 10https://gerrit.wikimedia.org/r/252390 (owner: 10Dzahn) [22:55:32] PROBLEM - citoid endpoints health on sca1001 is CRITICAL: /api is CRITICAL: Could not fetch url http://10.64.32.153:1970/api: Timeout on connection while downloading http://10.64.32.153:1970/api [22:56:24] (03PS2) 10Dzahn: irqbalance: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252391 [22:59:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [22:59:13] RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy [22:59:41] (03PS3) 10Dzahn: irqbalance: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252391 [23:03:34] (03CR) 10Dzahn: [C: 032] "minimal lint fix, just resource title quotes" [puppet] - 10https://gerrit.wikimedia.org/r/252391 (owner: 10Dzahn) [23:04:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [23:05:21] PROBLEM - Kafka Broker Replica Max Lag on kafka1013 is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [5000000.0] [23:09:02] PROBLEM - Kafka Broker Replica Max Lag on kafka1013 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [5000000.0] [23:09:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [23:14:11] PROBLEM - check_puppetrun on americium is CRITICAL: CRITICAL: Puppet has 1 failures [23:16:22] PROBLEM - Kafka Broker Replica Max Lag on kafka1013 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [5000000.0] [23:19:11] RECOVERY - check_puppetrun on americium is OK: OK: Puppet is currently enabled, last run 99 seconds ago with 0 failures [23:20:47] godog: I suppose the first thing for MW/swift is to make sure PrivateSettings.php is up to date with conf for codfw [23:23:31] PROBLEM - citoid endpoints health on sca1001 is CRITICAL: /api is CRITICAL: Could not fetch url http://10.64.32.153:1970/api: Timeout on connection while downloading http://10.64.32.153:1970/api [23:23:42] RECOVERY - Kafka Broker Replica Max Lag on kafka1013 is OK: OK: Less than 1.00% above the threshold [1000000.0] [23:24:38] (03PS2) 10Dzahn: role/lvs/balancer: double quoted strings [puppet] - 10https://gerrit.wikimedia.org/r/252392 [23:25:12] RECOVERY - citoid endpoints health on sca1001 is OK: All endpoints are healthy [23:26:48] (03PS2) 10Ori.livneh: Disable ChronologyProtector for rpc/RunJobs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252878 (owner: 10Aaron Schulz) [23:26:54] (03CR) 10Ori.livneh: [C: 032] Disable ChronologyProtector for rpc/RunJobs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252878 (owner: 10Aaron Schulz) [23:27:09] (03CR) 10Dzahn: [C: 032] "minimal and compiled: http://puppet-compiler.wmflabs.org/1281/" [puppet] - 10https://gerrit.wikimedia.org/r/252392 (owner: 10Dzahn) [23:27:21] (03Merged) 10jenkins-bot: Disable ChronologyProtector for rpc/RunJobs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/252878 (owner: 10Aaron Schulz) [23:30:56] !log ori@tin Synchronized rpc/RunJobs.php: I5e15ec9fb: Disable ChronologyProtector for rpc/RunJobs.php (duration: 00m 30s) [23:31:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:35:45] (03PS1) 10Dzahn: puppet-tests: remove broken inclusion of nagios.pp [puppet] - 10https://gerrit.wikimedia.org/r/253052 [23:37:41] (03CR) 10Dzahn: "jenkins FAIL should be fixed by: https://gerrit.wikimedia.org/r/#/c/253052/" [puppet] - 10https://gerrit.wikimedia.org/r/249655 (owner: 10Dzahn) [23:40:57] (03CR) 10Dzahn: base: lint fixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/249666 (owner: 10Dzahn) [23:56:16] (03PS2) 10Dzahn: base: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/249666 [23:57:23] (03CR) 10Dzahn: [C: 032] "and compiled http://puppet-compiler.wmflabs.org/1282/" [puppet] - 10https://gerrit.wikimedia.org/r/249666 (owner: 10Dzahn)