[00:14:57] (03PS1) 10Tim Landscheidt: Tools: Install libcgi-fast-perl [operations/puppet] - 10https://gerrit.wikimedia.org/r/147866 (https://bugzilla.wikimedia.org/68269) [00:22:03] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:35] ^^ [00:24:22] gitblit sucks [00:24:30] haha [00:24:39] use github if you need that sort of thing [00:24:43] it does a lot better job [00:24:53] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 53291 bytes in 0.372 second response time [01:10:26] (03PS1) 10Krinkle: docroot/default: Add width/height attributes to avoid reflow [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147869 [01:10:57] (03PS2) 10Krinkle: docroot/default: Add width/height attributes to img to fix reflow [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147869 [01:36:37] (03CR) 10Ori.livneh: [C: 032] docroot/default: Add width/height attributes to img to fix reflow [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147869 (owner: 10Krinkle) [01:36:44] (03Merged) 10jenkins-bot: docroot/default: Add width/height attributes to img to fix reflow [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147869 (owner: 10Krinkle) [01:37:50] ori: you deploying? [01:38:06] just did [01:38:24] really? [01:38:39] yes [01:38:40] right. i see it on view-source:http://wikimedia.community/s [01:38:45] but no log msg? [01:39:08] it stalled so i ctrl+breaked [01:39:08] Is sync-docroot no longer messaging logbot? [01:39:13] it ought to [01:39:48] I always wondered why (at least the old sync scripts) did the message at the end instead of the start [01:40:07] !log synced docroot/default/index.html (I005f43b96: Add width/height attributes to img to fix reflow) [01:40:10] when two people are thinking about doing something, it'd be useful to see it ahead of time, not afterwards [01:40:16] Logged the message, Master [01:40:32] or if someone's connection drops halfway through [01:40:53] yeah, though ideally it'd have a teardown to at least make the message come through [01:41:07] afterwards does allow "duration:". That's new though [01:42:51] by the way, re: https://gerrit.wikimedia.org/r/#/c/146681/ [01:43:18] it's weird to have a 'state' property that says 'ready' and also a 'skipped' property [01:43:26] module.state should be 'skipped' [01:43:39] though that's a bigger refactor i suppose [01:44:10] ori: Well, it depends. Do you know what skip does? [01:44:25] yes :) [01:44:41] maybe module.state = ['ready', 'skipped'] ? [01:44:42] It should not be too visible to the outside as to avoid people from being able to misuse it [01:45:02] resolve/ready is the accurate canonical state for the dependency has been resolved. [01:45:15] it just didn't require fetching a file, it did execute the function. [01:46:13] and getState is public, so we'd have to hide it from there I guess. Or do it in v2 as a breaking change. [01:47:08] ori: I also considered making .skip tristate, maybe you prefer that? [01:47:22] e.g. null/string as unprocessed, false/true as processed and its boolean result? [01:47:38] it saves a propery, but complicates usage. I liked it better separte. [01:47:40] separate. [01:47:56] maybe, but you shouldn't pander to my whims :) i haven't looked at javascript code in a while. [01:48:07] separate seems fine [02:15:50] !log LocalisationUpdate completed (1.24wmf13) at 2014-07-20 02:14:47+00:00 [02:16:00] Logged the message, Master [02:26:57] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-20 02:25:54+00:00 [02:27:03] Logged the message, Master [02:31:09] (03PS1) 10Ori.livneh: mediawiki::jobrunner: deduplicate job groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/147871 [02:59:05] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 20 02:57:59 UTC 2014 (duration 57m 58s) [02:59:11] Logged the message, Master [03:12:42] PROBLEM - mailman list info on sodium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:02] PROBLEM - HTTPS on sodium is CRITICAL: SSL_CERT CRITICAL: Error: verify depth is 6 [03:13:33] RECOVERY - mailman list info on sodium is OK: HTTP OK: HTTP/1.1 200 OK - 10533 bytes in 0.092 second response time [03:13:43] RECOVERY - HTTPS on sodium is OK: SSL_CERT OK - X.509 certificate for lists.wikimedia.org from RapidSSL CA valid until Jan 31 02:58:36 2016 GMT (expires in 560 days) [06:28:52] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:52] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:53] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:02] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:33] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:43] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:52] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:02] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:42] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:45:53] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:46:32] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:43] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:47:02] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:47:02] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:47:53] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:48:12] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki::multimedia: install libav-tools instead of ffmpeg on trusty [operations/puppet] - 10https://gerrit.wikimedia.org/r/147841 (owner: 10Ori.livneh) [07:49:05] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: only provision ini files for igbinary and wmerrors on precise [operations/puppet] - 10https://gerrit.wikimedia.org/r/147832 (owner: 10Ori.livneh) [07:56:36] (03PS1) 10Aude: set useRedirectTargetColumn setting to false for Wikibase [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147888 [08:50:49] (03CR) 10Aaron Schulz: [C: 031] mediawiki::jobrunner: deduplicate job groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/147871 (owner: 10Ori.livneh) [11:13:22] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 09:13:07 UTC [11:53:12] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jul 20 11:53:10 UTC 2014 [12:01:52] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 1 failures [12:19:52] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:17:45] (03PS1) 10Matanya: puppetmaster: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/147900 [13:41:52] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [13:55:19] (03PS1) 10Matanya: nginx: qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/147902 [13:58:52] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:11:38] (03PS1) 10Vogone: Added mrwiki to commonsuploads.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147906 (https://bugzilla.wikimedia.org/68292) [17:03:12] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:52] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [17:45:52] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [17:46:01] (03PS1) 10Amire80: Open Special:ContentTranslation in the target wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147918 [17:46:49] (03PS2) 10Ori.livneh: osmium(hhvm test host): remove ::hhvm::dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/147840 [17:47:43] (03CR) 10Ori.livneh: [C: 032 V: 032] "only touches osmium" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147840 (owner: 10Ori.livneh) [17:53:42] PROBLEM - DPKG on osmium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [17:54:43] RECOVERY - DPKG on osmium is OK: All packages OK [17:55:12] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:18:44] (03PS1) 10Bartosz Dziewoński: Set $wgCategoryCollation to 'uca-hr' on shwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147922 (https://bugzilla.wikimedia.org/67287) [18:41:53] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [18:55:02] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 53265 bytes in 0.272 second response time [19:00:52] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:19:05] (03PS1) 10Yuvipanda: labs_vagrant: Ensure proper ownership of vagrant homedir [operations/puppet] - 10https://gerrit.wikimedia.org/r/147982 [19:32:09] (03CR) 10Physikerwelt: [C: 031] labs_vagrant: Ensure proper ownership of vagrant homedir [operations/puppet] - 10https://gerrit.wikimedia.org/r/147982 (owner: 10Yuvipanda) [19:33:59] (03CR) 10JanZerebecki: [C: 031] blog -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/147739 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [19:37:54] (03CR) 10BryanDavis: labs_vagrant: Ensure proper ownership of vagrant homedir (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147982 (owner: 10Yuvipanda) [19:39:24] (03CR) 10Yuvipanda: labs_vagrant: Ensure proper ownership of vagrant homedir (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147982 (owner: 10Yuvipanda) [19:45:27] bd808|BUFFER: also, I wonder if having vagrant home *not* be on NFS would be a good idea [19:47:07] YuviPanda: I don't think it should really matter much. I think the usage of that directory is pretty limited. [19:51:50] bd808: hmm, true [20:28:52] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 1 failures [20:35:33] (03CR) 10JanZerebecki: [C: 031] ishmael -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/147740 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [20:39:08] (03PS2) 10Yuvipanda: labs_vagrant: Ensure proper ownership of vagrant homedir [operations/puppet] - 10https://gerrit.wikimedia.org/r/147982 [20:39:16] bd808: ^ udpdated. Once puppetception gets merged, I'll refactor this to use that [20:41:19] YuviPanda|zzz: It may be more prudent to make a new role that combines puppetception and mwv rather than breaking all 27 hosts using labs::vagrant at once. :) [20:41:34] bd808: why would it break? it can easily be backwards compatible [20:42:52] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [20:43:07] It can be as long as you move the exiting /srv/vagrant (that I broke for folks last week) to the new location rather than doing a new clone. [20:43:37] * bd808 is spooked after messing it up for everyone just a week ago [20:43:42] heh, that's true [20:43:55] maybe I should leave it alone and just use puppetception for its true, nefarious purpose... [20:44:06] which is https://meta.wikimedia.org/wiki/Research:Ideas/Public_query_interface_for_Labs [20:45:10] YuviPanda|zzz: Just install phpmyadmin :P [20:45:21] * bd808 knows that it a horrible idea [20:45:26] bd808: :P someone did that, and it was promptly shut down :P [20:45:31] (someone was not me, but Dispenser) [20:45:35] and for good reasoan [20:45:50] bd808: I also setup an anonymous mysql proxy, but that was also shut down :) [20:46:18] (with good reason) [20:48:03] YuviPanda|zzz: promptly being the time it took Coren to go 'bad and it's gone' :p [20:48:13] JohnLewis: indeed :) [20:48:51] this plan 1. respects labs ToS, 2. Prevents DDoS, 3. Is more userfriendly than phpmyadmin, 4. produces a public stream of SQL queries that can help grow our research community! [20:49:50] (you can't prevent ddos) [20:50:09] well, mitigate [20:50:16] Jasper_Deng: or, to be more exact [20:50:25] Jasper_Deng: prevent DDoS against *LabsDB*. DDoS against the service itself is ok [20:50:38] just don't pass the DDoS on [20:51:10] Hm. Love to see the more userfriendly part YuviPanda :) [20:51:18] :D [20:51:43] * JohnLewis reads the page to learn more [20:51:52] :D [20:52:03] I'm actually writing up the frontend for it right now. [20:52:15] and have a designer friend (prtksxna) who has offered to help [20:52:21] YuviPanda: lovely [20:52:36] JohnLewis: if this is a project you think would be useful, please +1 or something on the wiki page :) [20:52:50] YuviPanda: I will once I've read it :) [20:52:54] :D [20:52:57] ty [20:53:25] in return you +1 one of my future insane idea YuviPanda >:D [20:53:31] :D [20:55:28] YuviPanda: added - see my edit summary :D [20:56:40] nice :D [21:18:13] (03CR) 10Qgil: "This open changeset has almost two years! Should it be merged, updated, or abandoned?" [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/12147 (owner: 10QChris) [21:18:24] (03CR) 10Qgil: "This changeset has almost two years! Should it be merged, updated, or abandoned?" [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/12153 (owner: 10QChris) [21:21:34] (03Abandoned) 10QChris: Repairing status for items with toBeRun false. [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/12147 (owner: 10QChris) [21:21:42] (03Abandoned) 10QChris: Correcting md5 copying and item updating order [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/12153 (owner: 10QChris) [21:27:58] JohnLewis: I've a just-css prototype running on http://yuvi.in:5000/ [21:28:03] login doesn't work yet. [21:28:09] nothing works, but you get an idea [21:28:16] try 'new query' and 'list queries' [21:28:21] (which have dummy data, of course) [21:28:34] +1 for 'Because SQL doesn't need SSH' :p [21:28:50] :D [21:29:16] you could also 'Because SQL != SSH' :p [21:29:43] not pithy enough :) [21:29:58] Looks good so far though [21:30:11] good great [21:31:43] heh :D [21:31:51] now to actually *build* functionality... [21:32:05] ooh, I found the code on GitHub :p Starred and Followed! [21:32:09] :D [21:32:14] that was quick, was just going to point it to you [21:33:09] Well -:5000 and you just presented your GitHub repo then commits show data4all so, yeah [21:33:20] *GitHub account [21:34:14] SQL < SSH, clearly [21:35:05] :D [21:35:17] YuviPanda: When it's done - I might use it for a wikifarm I volunteer for; it'll be perfect for some of our stuff [21:35:22] :D [21:35:27] JohnLewis: orain? [21:35:32] Yeah :) [21:35:58] JohnLewis: heh, one of the *other* ulterior motives behind puppetception is that if someone wants to use ansible on labs, it should now be trivial to let them do so :) [21:36:03] of course, nobody has wanted to yet... [21:36:12] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Puppet has 2 failures [21:36:12] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 6 failures [21:36:12] PROBLEM - puppetmaster backend https on strontium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [21:36:13] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [21:36:16] JohnLewis: btw, I've an Extension:UrlShortener that orain might want to use :) [21:36:22] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 33 failures [21:36:27] I'll take a look at that now [21:36:31] it was in response to the url shortener RfC here, I need to push it on wikitech-l [21:36:32] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Puppet has 12 failures [21:36:33] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 58 failures [21:36:33] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 34 failures [21:36:42] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 31 failures [21:36:42] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 7 failures [21:36:42] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 24 failures [21:36:42] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 38 failures [21:36:42] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 38 failures [21:36:43] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 21 failures [21:36:52] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 31 failures [21:36:52] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 49 failures [21:36:52] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 61 failures [21:36:52] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 3 failures [21:36:56] JohnLewis: it's already being used at http://uncyclomedia.co/wiki/Special:UrlShortener [21:37:02] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 16 failures [21:37:02] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 25 failures [21:37:03] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet has 17 failures [21:37:12] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 57 failures [21:37:12] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 64 failures [21:37:12] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 15 failures [21:37:12] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 29 failures [21:37:12] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 29 failures [21:37:13] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 64 failures [21:37:13] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 19 failures [21:37:14] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 60 failures [21:37:14] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 34 failures [21:37:15] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 19 failures [21:37:15] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 16 failures [21:37:15] http://uncy.co/s/5 redirects to just that page, for example [21:37:16] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 40 failures [21:37:16] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Puppet has 24 failures [21:37:16] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 80 failures [21:37:22] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Puppet has 22 failures [21:37:22] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 22 failures [21:37:33] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 26 failures [21:37:33] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 54 failures [21:37:33] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 22 failures [21:37:33] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 60 failures [21:37:33] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 36 failures [21:37:34] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 56 failures [21:37:42] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 17 failures [21:37:42] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 17 failures [21:37:43] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 53 failures [21:37:43] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Puppet has 14 failures [21:37:43] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 24 failures [21:37:43] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 51 failures [21:37:43] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 61 failures [21:37:44] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 43 failures [21:37:44] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 58 failures [21:37:45] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 17 failures [21:37:45] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 55 failures [21:37:46] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 18 failures [21:37:52] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Puppet has 21 failures [21:37:52] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 18 failures [21:37:52] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 19 failures [21:37:52] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 27 failures [21:37:52] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 17 failures [21:37:53] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 15 failures [21:37:53] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 21 failures [21:37:53] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 21 failures [21:38:02] PROBLEM - puppet last run on db71 is CRITICAL: CRITICAL: Puppet has 22 failures [21:38:02] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 19 failures [21:38:11] ops ? [21:38:12] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 60 failures [21:38:12] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 18 failures [21:38:12] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 48 failures [21:38:12] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 26 failures [21:38:12] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 22 failures [21:38:12] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 60 failures [21:38:13] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 18 failures [21:38:14] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 25 failures [21:38:14] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 16 failures [21:38:22] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 58 failures [21:38:32] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 63 failures [21:38:33] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 22 failures [21:38:42] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 18 failures [21:38:42] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 21 failures [21:38:42] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 66 failures [21:38:42] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 21 failures [21:38:42] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 22 failures [21:38:43] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 34 failures [21:38:43] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 23 failures [21:38:44] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 58 failures [21:38:44] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 23 failures [21:38:45] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 58 failures [21:38:45] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 57 failures [21:38:46] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 43 failures [21:38:46] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 58 failures [21:38:47] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 22 failures [21:38:47] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 62 failures [21:38:48] wtf [21:38:51] heh [21:38:53] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 19 failures [21:38:53] PROBLEM - puppet last run on mexia is CRITICAL: CRITICAL: Puppet has 18 failures [21:38:53] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 21 failures [21:38:53] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 47 failures [21:38:53] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 18 failures [21:38:53] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 23 failures [21:38:54] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 21 failures [21:38:55] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 22 failures [21:38:55] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 20 failures [21:38:56] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 63 failures [21:39:02] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 16 failures [21:39:02] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 51 failures [21:39:02] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 44 failures [21:39:02] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 19 failures [21:39:02] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 24 failures [21:39:12] hoo: strontium sees to have died [21:39:12] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 21 failures [21:39:12] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 23 failures [21:39:12] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 24 failures [21:39:12] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 24 failures [21:39:12] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 54 failures [21:39:13] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 66 failures [21:39:13] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 18 failures [21:39:14] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 27 failures [21:39:14] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.055 second response time [21:39:22] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 55 failures [21:39:26] yeah, now they'll start unfailing [21:39:29] I guess [21:39:33] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 48 failures [21:39:33] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 43 failures [21:39:33] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 18 failures [21:39:42] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 17 failures [21:39:42] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 49 failures [21:39:43] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 58 failures [21:39:43] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 58 failures [21:39:43] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 47 failures [21:39:43] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 56 failures [21:39:43] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 24 failures [21:39:50] palladium doesn't seem too happy as well [21:39:52] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 67 failures [21:39:52] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 15 failures [21:39:52] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 66 failures [21:39:52] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 59 failures [21:39:53] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 29 failures [21:39:53] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 17 failures [21:39:53] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 22 failures [21:39:54] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 21 failures [21:40:04] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 57 failures [21:40:04] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 22 failures [21:40:04] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 24 failures [21:40:04] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 17 failures [21:40:12] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 24 failures [21:40:12] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 56 failures [21:40:12] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 61 failures [21:40:12] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:12] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 23 failures [21:40:13] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 17 failures [21:40:13] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 18 failures [21:40:13] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 18 failures [21:40:14] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 16 failures [21:40:22] hoo: can you see if strontium is alive ? [21:40:32] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 16 failures [21:40:33] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 55 failures [21:40:33] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:33] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 25 failures [21:40:42] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 61 failures [21:40:42] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 21 failures [21:40:42] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 23 failures [21:40:42] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 16 failures [21:40:42] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 65 failures [21:40:43] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:43] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 34 failures [21:40:44] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 53 failures [21:40:44] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:45] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 40 failures [21:40:45] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 59 failures [21:40:46] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 15 failures [21:40:52] PROBLEM - puppet last run on tantalum is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:52] matanya: it answers to ping [21:40:52] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:52] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 19 failures [21:40:53] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 30 failures [21:40:53] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 21 failures [21:40:53] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 20 failures [21:40:57] looking more [21:41:01] at least that :) [21:41:02] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 56 failures [21:41:03] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 20 failures [21:41:12] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 52 failures [21:41:12] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 15 failures [21:41:12] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 64 failures [21:41:12] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 64 failures [21:41:12] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 20 failures [21:41:13] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 43 failures [21:41:13] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 19 failures [21:41:14] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 64 failures [21:41:14] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 54 failures [21:41:15] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 18 failures [21:41:15] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 50 failures [21:41:16] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 22 failures [21:41:22] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 47 failures [21:41:32] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 59 failures [21:41:43] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 60 failures [21:41:43] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 24 failures [21:41:43] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 41 failures [21:41:43] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 54 failures [21:41:43] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 17 failures [21:41:43] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 17 failures [21:41:43] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 44 failures [21:41:52] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 25 failures [21:41:52] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 20 failures [21:41:53] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 17 failures [21:41:53] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 52 failures [21:42:02] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Puppet has 61 failures [21:42:03] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 60 failures [21:42:03] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 26 failures [21:42:12] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 20 failures [21:42:12] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 56 failures [21:42:12] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 20 failures [21:42:12] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 55 failures [21:42:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [21:42:13] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 21 failures [21:42:13] PROBLEM - puppet last run on es7 is CRITICAL: CRITICAL: Puppet has 15 failures [21:42:14] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 22 failures [21:42:14] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 27 failures [21:42:22] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 22 failures [21:42:22] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 19 failures [21:42:32] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 18 failures [21:42:33] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 61 failures [21:42:33] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 60 failures [21:42:42] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 25 failures [21:42:42] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 62 failures [21:42:43] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 18 failures [21:42:43] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 18 failures [21:42:43] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 56 failures [21:42:43] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 53 failures [21:42:43] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Puppet has 59 failures [21:42:44] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 21 failures [21:42:44] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 57 failures [21:42:45] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 17 failures [21:42:51] i'd leave it to ops to investigate [21:42:52] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 28 failures [21:42:52] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 14 failures [21:42:52] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 48 failures [21:42:53] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 17 failures [21:42:53] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 21 failures [21:42:53] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 53 failures [21:42:53] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Puppet has 21 failures [21:43:01] even total puppet failure won't bring down the cluster [21:43:02] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 60 failures [21:43:12] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 23 failures [21:43:12] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 20 failures [21:43:12] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 53 failures [21:43:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 2.454 second response time [21:43:13] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 21 failures [21:43:13] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 20 failures [21:43:19] ori: Try, but I can at least look ;) [21:43:22] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 21 failures [21:43:22] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 15 failures [21:43:25] * True [21:43:33] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Puppet has 21 failures [21:43:33] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 20 failures [21:43:33] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 18 failures [21:43:34] it's probably another transient failure with apache / mod_passenger on the puppetmasters [21:43:42] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet has 65 failures [21:43:43] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 25 failures [21:43:43] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 15 failures [21:43:43] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 24 failures [21:43:43] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 23 failures [21:43:43] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet has 53 failures [21:43:43] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Puppet has 42 failures [21:43:44] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 19 failures [21:43:44] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 62 failures [21:43:47] i think i dug up the cause on some mailing list [21:43:52] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 30 failures [21:43:52] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 24 failures [21:43:53] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 17 failures [21:43:53] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 63 failures [21:43:53] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 22 failures [21:43:53] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 22 failures [21:43:53] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 57 failures [21:44:02] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 55 failures [21:44:03] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 52 failures [21:44:03] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 20 failures [21:44:03] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 19 failures [21:44:03] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 22 failures [21:44:12] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 59 failures [21:44:12] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 55 failures [21:44:12] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 18 failures [21:44:12] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet has 54 failures [21:44:12] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 23 failures [21:44:13] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 26 failures [21:44:13] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 21 failures [21:44:24] ori: looks like a network issue: https://ganglia.wikimedia.org/latest/graph_all_periods.php?h=strontium.eqiad.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1405892613&g=network_report&z=large&c=Miscellaneous%20eqiad [21:44:32] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 14 failures [21:44:33] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 18 failures [21:44:33] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 17 failures [21:44:33] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 49 failures [21:44:39] matanya: No [21:44:42] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 17 failures [21:44:43] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 64 failures [21:44:43] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 50 failures [21:44:43] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 23 failures [21:44:43] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 51 failures [21:44:43] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 20 failures [21:44:43] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: Puppet has 20 failures [21:44:44] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 58 failures [21:44:44] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 56 failures [21:44:45] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 57 failures [21:44:45] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 56 failures [21:44:45] strontium is serving 500s [21:44:46] oh, cpu dropped too: https://ganglia.wikimedia.org/latest/graph_all_periods.php?h=strontium.eqiad.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1405892613&g=cpu_report&z=large&c=Miscellaneous%20eqiad [21:44:52] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 20 failures [21:44:52] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 23 failures [21:44:52] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 20 failures [21:44:52] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 20 failures [21:44:52] 500s are lightweight [21:45:03] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 24 failures [21:45:12] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 32 failures [21:45:12] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 23 failures [21:45:12] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 62 failures [21:45:12] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 25 failures [21:45:12] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 69 failures [21:45:13] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 20 failures [21:45:13] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 17 failures [21:45:13] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 51 failures [21:45:14] PROBLEM - puppet last run on db73 is CRITICAL: CRITICAL: Puppet has 18 failures [21:45:14] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 21 failures [21:45:15] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 21 failures [21:45:18] makes sense [21:45:22] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 26 failures [21:45:22] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 72 failures [21:45:22] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Puppet has 19 failures [21:45:22] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 17 failures [21:45:22] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 54 failures [21:45:23] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 23 failures [21:45:33] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 18 failures [21:45:33] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Puppet has 26 failures [21:45:33] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 22 failures [21:45:42] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 20 failures [21:45:42] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: Puppet has 18 failures [21:45:42] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 19 failures [21:45:42] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 59 failures [21:45:43] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 20 failures [21:45:43] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 54 failures [21:45:43] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 12 failures [21:45:44] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 54 failures [21:45:44] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 20 failures [21:45:52] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 54 failures [21:45:52] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Puppet has 14 failures [21:45:52] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 67 failures [21:45:53] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 22 failures [21:45:53] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 14 failures [21:45:53] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 25 failures [21:45:53] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 17 failures [21:45:54] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 51 failures [21:45:54] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 16 failures [21:45:55] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 18 failures [21:45:55] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 21 failures [21:45:56] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Puppet has 21 failures [21:46:02] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet has 20 failures [21:46:02] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 25 failures [21:46:03] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 22 failures [21:46:04] good luck ops. I'll file RT ticket, right ori? [21:46:12] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 57 failures [21:46:12] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 21 failures [21:46:12] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 24 failures [21:46:12] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 22 failures [21:46:12] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Puppet has 20 failures [21:46:22] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 14 failures [21:46:22] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:22] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 22 failures [21:46:22] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 21 failures [21:46:22] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 63 failures [21:46:23] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 18 failures [21:46:25] matanya: sure, better than nothign [21:46:31] doing [21:46:32] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 15 failures [21:46:33] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Puppet has 19 failures [21:46:33] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 63 failures [21:46:33] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 27 failures [21:46:42] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 55 failures [21:46:42] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Puppet has 44 failures [21:46:42] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 23 failures [21:46:43] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 47 failures [21:46:43] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 66 failures [21:46:43] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 21 failures [21:46:43] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 21 failures [21:46:52] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Puppet has 14 failures [21:46:52] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 53 failures [21:46:53] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 9 failures [21:46:53] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 19 failures [21:46:53] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 56 failures [21:46:53] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 51 failures [21:46:53] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 53 failures [21:46:54] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 19 failures [21:47:02] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 57 failures [21:47:12] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 16 failures [21:47:12] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 62 failures [21:47:12] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 46 failures [21:47:12] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 47 failures [21:47:12] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 21 failures [21:47:13] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 20 failures [21:47:13] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 24 failures [21:47:32] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 52 failures [21:47:32] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 57 failures [21:47:32] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 59 failures [21:47:33] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 22 failures [21:47:33] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 22 failures [21:47:33] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 59 failures [21:47:33] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 19 failures [21:47:42] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 61 failures [21:47:42] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 24 failures [21:47:42] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 58 failures [21:47:42] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 59 failures [21:47:42] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 61 failures [21:47:43] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 56 failures [21:47:43] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 21 failures [21:47:43] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 17 failures [21:47:44] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 59 failures [21:47:45] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 22 failures [21:47:52] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 58 failures [21:47:52] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 21 failures [21:47:52] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 55 failures [21:47:53] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 22 failures [21:47:53] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 56 failures [21:48:02] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 49 failures [21:48:02] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 20 failures [21:48:02] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 65 failures [21:48:12] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 60 failures [21:48:12] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 48 failures [21:48:12] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 58 failures [21:48:12] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 37 failures [21:48:12] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 52 failures [21:48:13] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 20 failures [21:48:13] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 23 failures [21:48:14] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 64 failures [21:48:14] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 60 failures [21:48:15] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 23 failures [21:48:15] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 22 failures [21:48:42] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 19 failures [21:48:42] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 26 failures [21:48:42] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 47 failures [21:48:42] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 19 failures [21:48:43] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 22 failures [21:48:43] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 58 failures [21:48:43] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 17 failures [21:48:52] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 60 failures [21:48:52] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 20 failures [21:48:52] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 48 failures [21:48:53] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 45 failures [21:48:53] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 54 failures [21:48:53] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 18 failures [21:48:53] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 68 failures [21:49:02] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 28 failures [21:49:02] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 26 failures [21:49:03] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 31 failures [21:49:03] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 22 failures [21:49:12] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Puppet has 25 failures [21:49:12] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 19 failures [21:49:12] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 53 failures [21:49:12] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 51 failures [21:49:12] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 43 failures [21:49:13] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 20 failures [21:49:13] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 26 failures [21:49:14] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 57 failures [21:49:14] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 19 failures [21:49:15] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 34 failures [21:49:15] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 21 failures [21:49:22] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 20 failures [21:49:22] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 60 failures [21:49:22] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 53 failures [21:49:22] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 18 failures [21:49:22] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 21 failures [21:49:32] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 18 failures [21:49:33] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 21 failures [21:49:33] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 23 failures [21:49:33] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 49 failures [21:49:33] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet has 16 failures [21:49:42] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 23 failures [21:49:43] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 24 failures [21:49:43] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 61 failures [21:49:43] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Puppet has 52 failures [21:49:43] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 59 failures [21:49:43] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 55 failures [21:49:43] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Puppet has 34 failures [21:49:44] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 20 failures [21:49:44] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 47 failures [21:49:45] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 45 failures [21:49:45] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 68 failures [21:49:46] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 18 failures [21:49:46] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 19 failures [21:49:52] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 21 failures [21:49:52] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 59 failures [21:49:52] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 22 failures [21:49:53] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 21 failures [21:49:53] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 23 failures [21:49:53] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 24 failures [21:49:53] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 21 failures [21:50:02] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 41 failures [21:50:12] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 64 failures [21:50:12] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 65 failures [21:50:12] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 16 failures [21:50:12] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Puppet has 23 failures [21:50:12] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 26 failures [21:50:13] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 22 failures [21:50:13] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 20 failures [21:50:14] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 21 failures [21:50:14] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 21 failures [21:50:15] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 60 failures [21:50:22] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 19 failures [21:50:22] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 29 failures [21:50:32] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 21 failures [21:50:33] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 18 failures [21:50:33] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 22 failures [21:50:33] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 18 failures [21:50:33] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: Puppet has 23 failures [21:50:42] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet has 18 failures [21:50:43] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 30 failures [21:50:43] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 23 failures [21:50:43] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 63 failures [21:50:43] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 45 failures [21:50:43] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 67 failures [21:50:43] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 18 failures [21:50:44] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 24 failures [21:50:44] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 21 failures [21:50:45] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 20 failures [21:50:52] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 21 failures [21:50:52] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 14 failures [21:50:53] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Puppet has 18 failures [21:50:53] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 53 failures [21:50:53] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet has 45 failures [21:51:03] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Puppet has 50 failures [21:51:12] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 23 failures [21:51:12] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Puppet has 45 failures [21:51:12] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 15 failures [21:51:12] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 17 failures [21:51:12] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 46 failures [21:51:13] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 55 failures [21:51:13] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 23 failures [21:51:14] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 53 failures [21:51:14] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Puppet has 16 failures [21:51:15] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet has 53 failures [21:51:15] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 24 failures [21:51:16] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 16 failures [21:51:16] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 24 failures [21:51:17] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 20 failures [21:51:17] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 29 failures [21:51:17] maybe someone should silence the alert? :P [21:51:18] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: Puppet has 44 failures [21:51:22] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 58 failures [21:51:22] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 22 failures [21:51:22] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 52 failures [21:51:32] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 15 failures [21:51:33] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 58 failures [21:51:33] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 24 failures [21:51:33] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Puppet has 28 failures [21:51:33] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 16 failures [21:51:33] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 55 failures [21:51:42] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Puppet has 24 failures [21:51:42] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Puppet has 58 failures [21:51:43] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 66 failures [21:51:43] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Puppet has 18 failures [21:51:43] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Puppet has 58 failures [21:51:43] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 27 failures [21:51:43] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 58 failures [21:51:44] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Puppet has 14 failures [21:51:44] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet has 52 failures [21:51:45] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 17 failures [21:51:45] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Puppet has 12 failures [21:51:46] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 26 failures [21:51:52] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 24 failures [21:51:52] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 17 failures [21:51:52] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 17 failures [21:51:53] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 21 failures [21:51:53] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 20 failures [21:51:53] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 63 failures [21:51:53] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 67 failures [21:51:54] PROBLEM - puppet last run on db69 is CRITICAL: CRITICAL: Puppet has 19 failures [21:52:02] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: Puppet has 24 failures [21:52:02] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 49 failures [21:52:02] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 15 failures [21:52:03] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 34 failures [21:52:03] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Puppet has 24 failures [21:52:12] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 65 failures [21:52:12] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 15 failures [21:52:12] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 31 failures [21:52:12] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 39 failures [21:52:12] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 52 failures [21:52:12] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Puppet has 22 failures [21:52:13] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 18 failures [21:52:14] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 25 failures [21:52:14] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Puppet has 47 failures [21:52:15] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.196 second response time [21:52:15] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 24 failures [21:52:16] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 22 failures [21:52:16] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 20 failures [21:52:22] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 58 failures [21:52:22] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 20 failures [21:52:22] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 22 failures [21:52:22] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Puppet has 21 failures [21:52:22] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 27 failures [21:52:33] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 18 failures [21:52:42] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: Puppet has 18 failures [21:52:42] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 24 failures [21:52:42] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 57 failures [21:52:42] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Puppet has 57 failures [21:52:42] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Puppet has 21 failures [21:52:43] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 55 failures [21:52:43] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 60 failures [21:52:44] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet has 51 failures [21:52:44] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 24 failures [21:52:45] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 56 failures [21:52:45] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 55 failures [21:52:46] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet has 21 failures [21:52:52] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet has 18 failures [21:52:52] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 33 failures [21:52:52] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet has 16 failures [21:52:52] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 59 failures [21:52:53] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 16 failures [21:52:53] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 52 failures [21:53:02] PROBLEM - puppet last run on db72 is CRITICAL: CRITICAL: Puppet has 20 failures [21:53:03] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Puppet has 24 failures [21:53:03] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 22 failures [21:53:03] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet has 18 failures [21:53:03] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 55 failures [21:53:12] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 59 failures [21:53:12] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 20 failures [21:53:12] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 59 failures [21:53:12] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet has 19 failures [21:53:12] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Puppet has 16 failures [21:53:13] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 63 failures [21:53:13] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: Puppet has 23 failures [21:53:14] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Puppet has 23 failures [21:53:22] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: Puppet has 15 failures [21:53:22] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 21 failures [21:53:22] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Puppet has 22 failures [21:53:32] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 35 failures [21:53:32] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 22 failures [21:53:33] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 24 failures [21:53:42] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Puppet has 40 failures [21:53:42] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 14 failures [21:53:42] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 19 failures [21:53:43] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Puppet has 22 failures [21:53:43] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 52 failures [21:53:43] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Puppet has 62 failures [21:53:43] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: Puppet has 37 failures [21:53:44] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 26 failures [21:53:44] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet has 22 failures [21:53:52] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet has 16 failures [21:53:52] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Puppet has 64 failures [21:53:52] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Puppet has 23 failures [21:53:52] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 15 failures [21:53:52] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 23 failures [21:53:53] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 22 failures [21:53:53] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 51 failures [21:53:54] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet has 21 failures [21:54:02] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 26 failures [21:54:02] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Puppet has 24 failures [21:54:03] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 26 failures [21:54:12] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 27 failures [21:54:12] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Puppet has 18 failures [21:54:12] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Puppet has 45 failures [21:54:12] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 53 failures [21:54:12] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 29 failures [21:54:13] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 26 failures [21:54:13] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 21 failures [21:54:13] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Puppet has 17 failures [21:54:22] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 19:53:17 UTC [21:54:22] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Puppet has 20 failures [21:54:22] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 48 failures [21:54:32] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 28 failures [21:54:33] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 66 failures [21:54:33] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Puppet has 18 failures [21:54:42] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet has 22 failures [21:54:42] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 63 failures [21:54:42] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 61 failures [21:54:42] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 48 failures [21:54:43] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 54 failures [21:54:43] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 52 failures [21:54:43] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 62 failures [21:54:44] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 52 failures [21:54:44] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 25 failures [21:54:44] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 13 failures [21:54:45] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 53 failures [21:54:52] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 12 failures [21:54:52] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 62 failures [21:54:52] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet has 53 failures [21:55:02] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 53 failures [21:55:02] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet has 14 failures [21:55:03] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet has 20 failures [21:55:03] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet has 19 failures [21:55:12] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Puppet has 47 failures [21:55:12] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 67 failures [21:55:13] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 44 failures [21:55:13] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 28 failures [21:55:13] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [21:55:13] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Puppet has 13 failures [21:55:13] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 60 failures [21:55:32] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 56 failures [21:55:42] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 22 failures [21:55:42] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Puppet has 22 failures [21:55:43] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 18 failures [21:55:53] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 17 failures [21:55:53] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 13 failures [21:56:02] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 20 failures [21:56:02] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 13 failures [21:56:02] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 21 failures [21:56:03] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 19 failures [21:56:12] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Puppet has 22 failures [21:56:12] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 22 failures [21:56:12] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Puppet has 43 failures [21:56:12] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 18 failures [21:56:12] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 19 failures [21:56:13] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.017 second response time [21:56:42] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 17 failures [21:56:42] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 25 failures [21:56:42] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 17 failures [21:57:13] :/ [21:57:33] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 21 failures [21:57:43] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 20 failures [21:58:22] (03PS1) 10Ori.livneh: mediawiki::multimedia: on trusty, install libvips37 rather than libvips15 [operations/puppet] - 10https://gerrit.wikimedia.org/r/147992 [22:00:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [22:05:12] PROBLEM - puppet last run on tridge is CRITICAL: CRITICAL: Complete puppet failure [22:05:42] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Complete puppet failure [22:06:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.028 second response time [22:07:31] (03PS1) 10Ori.livneh: mediawiki: don't attempt to load mod_version on Trusty [operations/puppet] - 10https://gerrit.wikimedia.org/r/147994 [22:09:42] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Complete puppet failure [22:10:57] oh ori BTW, did you reply to lior, the guy from zend ? [22:13:22] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jul 20 22:13:21 UTC 2014 [22:21:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [22:24:12] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:24:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 3.830 second response time [22:28:13] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [22:29:12] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Complete puppet failure [22:31:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.151 second response time [22:38:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [22:38:22] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 20 Jul 2014 20:37:30 UTC [22:39:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.036 second response time [22:40:02] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Complete puppet failure [22:43:42] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Complete puppet failure [22:44:42] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:45:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [22:48:12] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:48:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 4.190 second response time [22:48:42] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:01:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:01:42] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:02:13] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 3.009 second response time [23:20:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:23:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.031 second response time [23:29:52] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Complete puppet failure [23:32:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:34:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.416 second response time [23:34:59] (03PS2) 10Ori.livneh: mediawiki::web: compatability fixes for apache2.conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/147994 [23:37:52] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Complete puppet failure [23:43:43] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Complete puppet failure [23:49:12] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Complete puppet failure [23:52:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:53:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.032 second response time [23:53:52] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Complete puppet failure [23:58:12] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:59:12] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.023 second response time