[00:00:04] Everyone does it once (or twice or ...). ;-) [00:00:14] sadly volapuk wikipedia was destroyed in the incident. [00:00:37] Eloquence: fabrice's last email from this afternoon said "Many thanks to Matthias, Sebastian and Denis for all their good work in today's deployment attempt on German Wikipedia! … Our goal is to deploy on both English and German Wikipedias this Tuesday" [00:01:08] looks like some broken communications :( mlitn any idea how this happened? [00:01:40] There is more information on the ops-l thread that was CCed to mlitn [00:01:56] i'm pretty sure that tuesday deploy won't happen now :) [00:02:24] It looks like the new database was missing indexes, making the queries very slow, making Apache threads block waiting for the DB, overwhelming the Apache machines with active threads, doom, end of the world, etc. [00:03:18] i hate what this extension is using for a primary key [00:03:33] RoanKattouw: So there was a deployment of new code without first deploying schema changes? [00:03:36] binasher: AFT was not completely disabled code-wise (data migration script running, slowly, for enwiki), but from a visitor-perspective it was not accessible on the site [00:03:58] mlitn: that doesn't seem to be the case, based on the comments in the queries [00:04:14] Susan: or... something. I wasn't paying attention, I don't know exactly what happened. But that seems to be what binasher was saying earlier [00:04:29] SELECT /* DataModelBackendLBFactory::getList DrKiernan */ ... [00:04:35] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54102 [00:04:46] SELECT /* ArticleFeedbackv5BackendLBFactory::getContributionsData 180.76.5.192 */ ... [00:04:58] mlitn: coming from all the apaches [00:05:36] had to be exposed to user facing pages, hit by lots of logged in users and anons [00:06:07] mycontribs hook [00:06:58] Susan: oh, no, the schema as designed is deployed.. it just.. uh.. doesn't scale [00:07:07] or fit the queries [00:07:19] Should've bought the Web scale® version. (o; [00:07:25] new aftv5, same as the old aftv5 ;) [00:07:28] (sorrt mlitn) [00:07:32] *sorry [00:08:36] Susan, unfortunately that requires MongoDB, which is also too awesome for us. [00:08:40] binasher: it's (unfortunately) backed by facts [00:09:05] the contribs hook does seem to query for unindexed columns aft_user & aft_user_text [00:09:51] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [00:10:00] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [00:10:27] a lot of these queries are a bit.. odd [00:10:43] * Susan thinks of other silly names to accompany MariaDB and MongoDB. Five letters and begins with an M is the pattern, I suppose. [00:11:08] SELECT aft_id,aft_page,"AFT" AS aft_contribution,aft_timestamp AS rev_timestamp … (what's the "AFT" ?) [00:11:16] "Article Feedback Tool" [00:12:14] Helpful, Coren. [00:12:22] SELECT *,aft_relevance_score AS offset_value FROM `aft_feedback` WHERE aft_page IN ('19133401','31441675','18996620','19137063','31458992','1606665','36739487','36789451','31969222','38235971','38235972','36789275','36661976','38235973','36862920','36794434','28464394','37622924','36845193','37597084') AND (aft_has_comment = 1) AND (aft_oversight = 0) AND (aft_archive = 0) AND (aft_hide = 0) AND (aft_feature = 0) AND (aft_resolve = 0) AND [00:12:23] (aft_noaction = 0) AND (aft_inappropriate = 0) ORDER BY aft_relevance_score ASC,aft_id ASC LIMIT [00:12:34] the select *, column included in the * amuses me [00:12:46] Susan: I never claimed I was useful. :-) [00:13:13] Ladies and gentlemen, the Wikimedia Foundation's newest hire! [00:13:27] * Susan gives Coren a hug. [00:13:30] I saw you had a pretty rough week. [00:13:31] Susan: I don't even think that's true. :-P [00:13:35] I almost wrote you a note. [00:13:45] Susan, at least he didn't get indef-blocked [00:13:47] Susan: Oh, you mean on enwp? Meh -- it's actually a serious load off my back. [00:13:51] Eloquence: Yet! [00:14:12] Also, it's amusing how many people know I was blocked. Somewhat amusing. [00:14:26] Coren|Rest: Sure, but it still seemed like the past few months hadn't been great for you. [00:14:37] binasher: it's a really hacky way to inject aft data into mycontributions: ContribsPages executes the quer(y|ies), AFT adds it's own bogus field "AFT" in there so that hook "ContributionsLineEnding" can intercept that data on build the output itself (rather than having contribspager do it, as that's focussed on pages/article entries) [00:14:42] Calling that job thankless doesn't really begin to capture it. [00:15:09] huh [00:15:23] Susan: The crap from without I can deal with; it's part of the job of an arb and you get used to it. Having to fight /in/ the committee, however, was something I could stand losing. And so, I did. [00:16:02] New patchset: Milimetric; "security fix for stat1 cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54105 [00:20:16] binasher: mycontribs is not really "open" to other data than what it was originally built for; e.g. "rev_timestamp" is hardcoded as "IndexField", etc. In order to get it's data in mycontributions, it also needs the column "rev_timestamp" (that's what contribspages will automatically sort the data by, if I'm not mistaken) [00:20:22] so that gets the data in [00:22:09] then, just before outputting a line, another hook "ContributionsLineEnding" is called, which could be used to change the exact text outputted per contribs entry; aft hooks into that, checks if that "line" (row) has a column "AFT"; if yes: build the output, if not, do nothing and let contribspages build it (since it is no AFT entry, it's a page entry) [00:23:06] probably quite an incomprehensible explanation? :p [00:23:59] that's no excuse for the missing index though; even though the mycontribs hook is rather hacky, it did work (if only the data is properly indexed) [00:24:05] New patchset: Milimetric; "security fix for stat1 cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54105 [00:27:11] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54105 [00:28:27] mlitn: would you mind also changing the id generator from the one in DataModel to the time-based uuid generator in core? it should be better ordered and have better concurrency [00:28:36] AaronSchulz will email details [00:30:00] PROBLEM - MySQL Replication Heartbeat on db67 is CRITICAL: CRIT replication delay 182 seconds [00:30:01] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 181 seconds [00:30:01] PROBLEM - MySQL Slave Delay on db63 is CRITICAL: CRIT replication delay 181 seconds [00:30:01] PROBLEM - MySQL Slave Delay on db60 is CRITICAL: CRIT replication delay 181 seconds [00:30:01] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 182 seconds [00:30:11] i definitely will [00:30:52] PROBLEM - MySQL Replication Heartbeat on db60 is CRITICAL: CRIT replication delay 182 seconds [00:33:50] RECOVERY - MySQL Replication Heartbeat on db60 is OK: OK replication delay 4 seconds [00:34:01] RECOVERY - MySQL Replication Heartbeat on db67 is OK: OK replication delay 0 seconds [00:34:01] RECOVERY - MySQL Slave Delay on db63 is OK: OK replication delay 0 seconds [00:34:01] RECOVERY - MySQL Slave Delay on db60 is OK: OK replication delay 0 seconds [00:48:01] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [01:15:30] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Sat Mar 16 01:15:28 UTC 2013 [01:25:34] New patchset: Catrope; "Temporarily install apachebench on wtp1004" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54110 [02:07:01] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [02:29:19] !log LocalisationUpdate completed (1.21wmf11) at Sat Mar 16 02:29:19 UTC 2013 [02:29:27] Logged the message, Master [02:52:04] PROBLEM - Puppet freshness on europium is CRITICAL: Puppet has not run in the last 10 hours [03:10:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [03:17:10] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [03:17:20] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 0 seconds [03:18:12] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [03:18:12] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [03:18:12] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [03:18:12] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [03:19:52] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 193 seconds [03:20:10] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 200 seconds [03:37:32] New review: Krinkle; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54073 [04:00:02] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [04:08:01] PROBLEM - Puppet freshness on sq83 is CRITICAL: Puppet has not run in the last 10 hours [04:08:02] PROBLEM - Puppet freshness on tmh1002 is CRITICAL: Puppet has not run in the last 10 hours [04:18:11] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.671 second response time [06:08:00] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [06:24:00] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [06:28:11] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 184 seconds [06:28:11] PROBLEM - MySQL Replication Heartbeat on db1039 is CRITICAL: CRIT replication delay 187 seconds [06:28:22] PROBLEM - MySQL Replication Heartbeat on db45 is CRITICAL: CRIT replication delay 193 seconds [06:28:51] PROBLEM - MySQL Replication Heartbeat on db1021 is CRITICAL: CRIT replication delay 227 seconds [06:28:51] PROBLEM - MySQL Replication Heartbeat on db1005 is CRITICAL: CRIT replication delay 227 seconds [06:29:01] PROBLEM - MySQL Replication Heartbeat on db55 is CRITICAL: CRIT replication delay 233 seconds [06:29:11] PROBLEM - MySQL Replication Heartbeat on db1026 is CRITICAL: CRIT replication delay 237 seconds [06:32:56] RECOVERY - MySQL Replication Heartbeat on db1021 is OK: OK replication delay 0 seconds [06:32:56] RECOVERY - MySQL Replication Heartbeat on db1005 is OK: OK replication delay 0 seconds [06:33:01] RECOVERY - MySQL Replication Heartbeat on db55 is OK: OK replication delay 0 seconds [06:33:01] RECOVERY - MySQL Replication Heartbeat on db1026 is OK: OK replication delay 0 seconds [06:33:10] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay 0 seconds [06:33:11] RECOVERY - MySQL Replication Heartbeat on db1039 is OK: OK replication delay 0 seconds [06:33:20] RECOVERY - MySQL Replication Heartbeat on db45 is OK: OK replication delay 0 seconds [07:17:07] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [07:17:22] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [07:21:40] come on, gerrit [07:21:43] New patchset: Ori.livneh; "Ensure Python deps for limnpy are on stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [07:21:47] thanks. [07:22:37] > Provides: python-yuvipanda (>=1.3.3.7) [07:22:40] sbm [07:23:15] :) [07:28:58] New patchset: Ori.livneh; "Ensure Python deps for limnpy are on stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [07:40:39] LeslieCarr, Ryan_Lane: https://www.wikimedia.org/ has been updated to include Labs. [09:07:30] New patchset: Ori.livneh; "Rsync public data for visualization to stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [09:12:20] New patchset: Ori.livneh; "Rsync public data for visualization to stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [09:29:30] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 183 seconds [09:30:11] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 196 seconds [09:39:10] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 13 seconds [09:39:30] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 1 seconds [10:11:20] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 185 seconds [10:11:30] PROBLEM - MySQL Slave Delay on db44 is CRITICAL: CRIT replication delay 185 seconds [10:36:31] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay 6 seconds [10:36:31] RECOVERY - MySQL Slave Delay on db44 is OK: OK replication delay 1 seconds [10:49:00] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [10:52:15] what deployment hit en.wiki last night? [10:52:47] something broke the course pages for the Education Program extension. :( [11:40:31] PROBLEM - MySQL Slave Delay on db32 is CRITICAL: CRIT replication delay 189 seconds [11:40:31] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 190 seconds [12:08:01] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [12:32:55] hello [12:33:38] !log restarting Jenkins on a quiet day to have it using umask 0002 (aka g+w). That needs a full restart since the umask is set by /usr/bin/daemon See {{gerrit|53990}}. [12:33:47] Logged the message, Master [12:35:04] New review: Hashar; "Restarted Jenkins right now since that is quiet :-] Sat, 16 Mar 2013 12:34:46 +0000" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53990 [12:43:14] quiet day?! [12:43:24] i throw a patch at you, hah [12:48:24] thanks ori! [12:50:54] np, the only special talent needed to fix simple bugs like this is to be the only one up :P [12:51:21] can you push it out too? or did you already start? [12:53:00] PROBLEM - Puppet freshness on europium is CRITICAL: Puppet has not run in the last 10 hours [12:53:31] RECOVERY - MySQL Slave Delay on db32 is OK: OK replication delay 12 seconds [12:53:31] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 12 seconds [12:54:28] more or less away while jenkins finish restarting [12:54:46] !log olivneh synchronized php-1.21wmf11/extensions/EducationProgram/includes/pages/EducationPage.php 'Fix fatal error in EducationProgram (Bug 46188)' [12:54:53] Logged the message, Master [12:55:25] ragesoss: fixed [12:55:38] sweeeeeeeeeeeet [12:55:42] thanks hashar [13:00:42] good night & good luck with the jenkins stuff [13:03:47] New review: Aude; "yes it's worth a try :) we also need some changes in the dispatcher script to optimize it there" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52797 [13:04:04] if any ops folks are around, https://gerrit.wikimedia.org/r/#/c/52797/ [13:04:17] a second cron job for dispatching wikidata changes [13:07:34] !log Mar 16, 2013 1:06:47 PM Jenkins is fully up and running [13:07:41] Logged the message, Master [13:19:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [13:19:01] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [13:19:01] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [13:19:01] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [13:24:01] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Sat Mar 16 13:23:57 UTC 2013 [13:36:57] New review: Hashar; "Not sure it is needed but why not. Make sure to get Timo trained by ops about the do and don't on a ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54073 [13:37:50] RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Sat Mar 16 13:37:43 UTC 2013 [14:01:10] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [14:40:40] !log reedy synchronized php-1.21wmf11/extensions/EducationProgram [14:40:45] Logged the message, Master [14:48:09] !log reedy synchronized php-1.21wmf11/extensions/EducationProgram [14:48:16] Logged the message, Master [14:57:50] !log reedy synchronized php-1.21wmf11/extensions/EducationProgram [14:57:59] Logged the message, Master [15:09:30] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 181 seconds [15:09:40] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 183 seconds [16:05:30] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [16:05:40] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [16:09:00] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [16:11:30] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 187 seconds [16:11:42] PROBLEM - MySQL Slave Delay on db44 is CRITICAL: CRIT replication delay 185 seconds [16:15:32] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 183 seconds [16:15:40] PROBLEM - MySQL Slave Delay on db44 is CRITICAL: CRIT replication delay 185 seconds [16:25:00] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [16:26:43] RECOVERY - MySQL Slave Delay on db44 is OK: OK replication delay 15 seconds [16:27:30] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay 1 seconds [17:22:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.629 second response time [17:34:40] PROBLEM - MySQL Slave Delay on db44 is CRITICAL: CRIT replication delay 185 seconds [17:35:40] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 206 seconds [17:47:42] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 186 seconds [17:47:42] PROBLEM - MySQL Slave Delay on db44 is CRITICAL: CRIT replication delay 191 seconds [17:56:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.818 second response time [18:04:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.656 second response time [18:27:40] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 185 seconds [18:27:41] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 185 seconds [18:31:40] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [18:31:41] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [18:38:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:41] PROBLEM - MySQL disk space on blondel is CRITICAL: DISK CRITICAL - free space: /a 11169 MB (3% inode=99%): [18:52:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.595 second response time [18:55:42] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay 0 seconds [18:55:44] RECOVERY - MySQL Slave Delay on db44 is OK: OK replication delay 0 seconds [19:25:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [20:10:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:31] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 181 seconds [20:15:40] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 183 seconds [20:25:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [20:38:31] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [20:38:41] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [20:50:00] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [20:53:31] New patchset: Matmarex; "(bug 39225) Configure $wgBabelCategoryNames and $wgBabelMainCategory for pl.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/54144 [22:07:01] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [22:09:00] PROBLEM - Puppet freshness on palladium is CRITICAL: Puppet has not run in the last 10 hours [22:37:00] PROBLEM - Puppet freshness on mw1061 is CRITICAL: Puppet has not run in the last 10 hours [22:51:40] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 194 seconds [22:51:50] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 197 seconds [22:54:00] PROBLEM - Puppet freshness on europium is CRITICAL: Puppet has not run in the last 10 hours [23:20:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [23:20:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [23:20:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [23:20:00] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours