[00:04:33] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [01:26:35] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:39:30] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:41:18] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 206 seconds [01:41:36] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 227 seconds [01:47:53] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 603s [01:54:39] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [01:58:15] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 11s [01:58:33] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 10 seconds [02:02:36] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [02:19:33] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [02:19:57] !log LocalisationUpdate completed (1.20wmf9) at Thu Aug 9 02:19:57 UTC 2012 [02:20:11] Logged the message, Master [02:33:29] my father had a toolserv-related question. The "CONTRIBUTORS" tool is lagged by 16 days; he wanted to know if it's reasonably expected to catch up any time soon. [02:33:31] anyone?> [02:34:17] you should link to it [02:34:25] and probably ask #wikimedia-toolserver [02:34:31] and bring him to a meetup [02:37:02] http://toolserver.org/~daniel/WikiSense/Contributors.php?wikilang=en&wikifam=.wikipedia.org&grouped=on&page=Acetylene [02:37:04] (for instance) [02:37:11] you'll note that it's lagged 16 days 4 hours [02:37:26] and I asked on the toolserv channel first [02:37:33] !log LocalisationUpdate completed (1.20wmf8) at Thu Aug 9 02:37:33 UTC 2012 [02:37:42] Logged the message, Master [02:41:36] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Thu Aug 9 02:41:13 UTC 2012 [02:42:38] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [02:57:30] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [03:10:05] RECOVERY - Puppet freshness on srv281 is OK: puppet ran at Thu Aug 9 03:09:45 UTC 2012 [03:12:20] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [03:12:38] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [03:27:29] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [03:37:05] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Thu Aug 9 03:36:46 UTC 2012 [04:06:32] Brooke: did you see my bugzilla bug? ;) [04:06:50] http://www.mail-archive.com/wikibugs-l@lists.wikimedia.org/msg152480.html [04:07:00] something's wrong with this picture [04:16:22] jeremyb: What's the bug? [04:16:31] Looks like a bug conflict or something? [04:19:01] Brooke: i cc'd myself and it mailed everyone on the bug except matanya [04:19:21] see also http://thread.gmane.org/gmane.org.wikimedia.mediawiki.bugs/222261/focus=223436 [04:19:27] matanya's comment is missing [04:19:47] matanya's comment was *definitely* already there when i submitted the form to CC [04:20:15] so it therefore wasn't a midair collision [04:20:44] https://bugzilla.wikimedia.org/show_bug.cgi?id=35578#c12 [04:20:47] The comment is there now. [04:21:06] I'm not sure I understand the anomaly, with the exception of a bit of strange BZ e-mail. [04:22:03] it wasn't just a strange email [04:22:08] it's also a missing email [04:22:18] look at the time difference. it was over a week! [04:23:28] I heard about other BZ e-mail strangeness recently, I think. [04:23:31] Chatter on IRC somewhere, maybe. [04:56:36] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [04:56:36] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [05:41:08] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [05:43:33] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [06:02:35] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [09:14:55] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [09:25:25] RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Thu Aug 9 09:24:53 UTC 2012 [09:27:28] hey Tim-away [09:28:52] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [09:28:52] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [09:28:52] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [09:32:57] * jubo2 has been banging head on wall trying to install Nagios .. nconf looks just great if it'd actually work till the end [09:33:29] if anyone knows of a better / more working utility then Nconf to configure Nagios pls let meknow [10:05:31] What does Wikimedia use for configuring Nagios ? [10:05:43] Puppett [10:05:56] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [10:30:16] So from googling I gather that puppet is system for managing server configurations [10:35:12] I cannot find info on puppet & nagios integration anywhere [10:35:15] it's not easier [10:35:31] not for a small system anyway [10:35:53] Tim-away: what do you suggest for configuring nagios ? [10:36:05] for how many servers? [10:36:18] Tim-away: we've got 2 [10:36:29] just write the configuration by hand [10:36:31] there's a manual [10:37:12] going to need a few more servers should once we reach the implementation phase the consumers find and embrace Consumium [10:37:48] if they don't, well,, there is not much I can do in that case [10:37:58] or 3x the marketing effort lol [10:40:32] It was a real heavy weight taken off my shoulders when Wikiteam took the action of backing http://develop.consumerium.org/wiki/ onto archive.org .. before that it was me and my neck on the line should backups fail and catastrophical wiki data loss encurs [10:40:56] that was 1.5 months ago [10:42:15] for over 10 yrs I needed to worry about looming data loss catastrophe should like the server hall and my flat and my friend's flat burn down simultaneously [10:43:01] that sounds like a pretty remote risk [10:47:18] I just yesterday finished 1st pass over all 2000+ written MW extensions evaluating them re Consumium [10:47:33] glad to hear that, im wikiteam founder ; ) [10:47:34] good stuff, good stuff [10:47:46] *hooray for Wikiteam* [10:50:33] jubo2, note that not all the extensions are quality ones [10:50:54] some would have security issues, or perhaps efficiency problems [10:50:57] Platonides: I've been noticing that [10:51:05] specially those pasted at the wiki :) [10:51:34] grrr [10:51:58] you know how wmerrors sets a timer to break deadlocks? [10:52:08] we welcome any patches to them, of course [10:52:16] I don't remember a timer in wmerrors [10:52:24] Platonides: I've been steering clear of those that store their code inwiki [10:52:28] http://paste.tstarling.com/p/dcrIwv.html [10:52:48] now it can't even *create* a timer without deadlocking [10:55:38] seems it's still at svn [10:57:27] looks you added the timer in 103433 [11:01:41] from the backtrace, timer_create also uses malloc() internally [11:02:41] it looks like I can call timer_create on RINIT and then timer_settime() from the signal handler [11:03:39] the problem is not timer creation but working without a good malloc() [11:04:03] how do oyu expect to make it solve the problem? [11:04:21] apparently, the only thing you could do is to exit() the process to release the mutex [11:04:39] or change the malloc() implementation [11:04:46] surely there must be some lockless one [11:06:00] abort(), not exit(), that's what it odes already [11:06:59] yep, we don't want exit() but something like _Exit() [11:07:05] none of the popular malloc implementations are signal-safe [11:07:22] the way I'm doing it already will work [11:08:30] it should work [11:18:04] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [11:20:10] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 1.62 ms [11:40:52] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [11:59:59] timstarling: all malloc implementations are signal safe, if you don't call malloc from within signal handler!!!11 [11:59:59] :) [12:00:32] PHP tries to run a whole new request from a signal handler [12:02:38] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [12:03:50] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [12:05:56] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [12:24:58] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [12:43:52] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [12:58:52] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [13:13:52] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [13:28:53] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [14:34:25] PROBLEM - Frontend Squid HTTP on cp1001 is CRITICAL: Connection refused [14:35:19] PROBLEM - Backend Squid HTTP on cp1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:43] PROBLEM - Host cp1001 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:28] RECOVERY - Host cp1001 is UP: PING OK - Packet loss = 0%, RTA = 35.50 ms [14:54:05] RECOVERY - Backend Squid HTTP on cp1001 is OK: HTTP OK HTTP/1.0 200 OK - 27407 bytes in 0.143 seconds [14:57:50] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [14:57:50] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [15:02:20] PROBLEM - check_all_memcacheds on spence is CRITICAL: MEMCACHED CRITICAL - Could not connect: 10.0.8.20:11000 (Connection timed out) [15:05:11] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [16:03:49] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [16:05:35] !log preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero' [16:05:44] Logged the message, Master [16:06:11] !log preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero' [16:06:20] Logged the message, Master [17:58:50] hello guys [17:58:58] is there a way to change your gerrit "Full name"? [18:00:32] <^demon> MatmaRex: No, it's not really possible right now. [18:01:11] eh [18:01:24] somebody set mine to "Matmarex" when i asked for "matmarex" and now i'm angry [18:02:38] ^demon: also, why isnt it possible? [18:02:57] there is a field in prefs, but it's disabled... [18:03:15] <^demon> Well, the reason the first letter is uppercased is it's the same as your wiki username (which is also first-letter-uppercased). [18:03:44] but there are people with all-lowercase name [18:03:45] e.g. saper [18:03:51] <^demon> Unfortunately, the changing-name thing doesn't work when you're using LDAP (this is a bug). If we changed it in LDAP, Gerrit would treat you as a new user and we'd have to manually merge them in the database. [18:04:41] oh well. okay then. [18:15:12] <^demon> MatmaRex: For what it's worth--I've had several people complain to me about this already, so it's likely to be something I'll look into fixing :) [18:15:35] ^demon: thanks [18:17:05] also another question: i'm wondering how long, approximately, i will have to wait before somebody looks at my submissions to gerrit? [18:20:05] MatmaRex: It's very variable, if you have a specific review in mind, then it's best to add them to it, then prod them from time to time [18:20:30] *reivwer [18:20:34] **reviewer [18:20:36] <^demon> Yes. Best thing to do is just add the person you want to review your work :) [18:20:58] i just submitted three quick fixes for live js preview [18:21:05] i dont need anybody in particular :P [18:21:31] (https://gerrit.wikimedia.org/r/#/dashboard/417) [18:52:00] bsitu: wmf8 is ready [19:15:49] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [19:17:56] !log catrope synchronized wmf-config/CommonSettings.php 'Unbreak links in VE' [19:18:05] Logged the message, Master [19:23:10] PROBLEM - Host db1011 is DOWN: PING CRITICAL - Packet loss = 100% [19:24:04] RECOVERY - Host db1011 is UP: PING OK - Packet loss = 0%, RTA = 35.90 ms [19:24:49] PROBLEM - Host db1009 is DOWN: PING CRITICAL - Packet loss = 100% [19:25:25] PROBLEM - Host db1010 is DOWN: PING CRITICAL - Packet loss = 100% [19:25:52] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [19:26:19] RECOVERY - Host db1009 is UP: PING OK - Packet loss = 0%, RTA = 35.37 ms [19:26:20] RECOVERY - Host db1010 is UP: PING OK - Packet loss = 0%, RTA = 35.40 ms [19:27:08] !log catrope synchronized wmf-config/CommonSettings.php [19:27:17] Logged the message, Master [19:28:06] about to run scap [19:28:22] hope it doesn't take an hour today :P [19:29:55] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [19:29:56] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [19:29:56] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [19:37:52] !log kaldari Started syncing Wikimedia installation... : [19:38:00] Logged the message, Master [19:48:34] !log catrope synchronized php-1.20wmf9/extensions/VisualEditor/ 'VE bugfix' [19:48:43] Logged the message, Master [20:06:50] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [20:09:30] !log kaldari Finished syncing Wikimedia installation... : [20:09:38] Logged the message, Master [20:26:45] kaldari: did you sync wmf8 or wmf9? [20:27:05] I ran scap which does both [20:27:27] the scap took forever though, so we still haven't gotten to the config changes [20:27:38] ah, gotcha. [20:31:35] kaldari: sometimes it's easier to do the config changes first [20:31:42] then just let stuff work with broken messages :p [20:32:23] RoanKattouw: our config change was switching how public the feature is, so we wanted to test the new version first [20:32:30] oops, that was for Reedy [20:33:17] RoanKattouw: We're about to deploy a config change and I see there is one live hack on fenari re $wgVisualEditorParsoidPrefix. Is that safe to push out? [20:33:54] Yes [20:33:59] I'll clean up the live hack later [20:34:14] thanks [20:40:12] binasher: http://en.wikipedia.org/wiki/Stat_%28system_call%29#Criticism_of_atime (I keep running into this page and snickering each time) [20:40:16] what is wmf-config/extension-list used for? [20:40:34] localisation cache syuff [20:41:10] AaronSchulz: hah, i like the quote in there [20:41:17] * binasher likes breaking posix compliance [20:41:55] Reedy: if I'm removing a deprecated extension, should extension-list be synced first, last, or doesn't matter? [20:42:35] shouldn't matter [20:44:04] thanks [20:46:19] RoanKattouw: It's saying that the live hack is going to be overwritten by the merge so I'm going to go ahead and commit your change if that's OK. [20:47:28] PROBLEM - MySQL disk space on db1011 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=86%): [20:48:58] RECOVERY - MySQL disk space on db1011 is OK: DISK OK [20:49:01] done [20:49:13] well, with that part anyway [20:52:14] !log kaldari synchronized wmf-config/CommonSettings.php 'Removing deprecated CustomUserSignup extension' [20:52:23] Logged the message, Master [20:52:31] !log kaldari synchronized wmf-config/InitialiseSettings.php 'Removing deprecated CustomUserSignup extension and turning on Curation Toolbar for en.wiki' [20:52:40] Logged the message, Master [20:52:49] !log kaldari synchronized wmf-config/extension-list 'Removing deprecated CustomUserSignup extension' [20:52:58] Logged the message, Master [20:53:21] ori-l: All yours now [20:53:45] kaldari: awesome, thanks [20:55:58] BTW, I got the following error while syncing: srv194: Host key verification failed. [20:56:05] not sure if it's important or not [20:56:24] no idea if srv194 is being used [20:57:51] It would seem it's supposed to be de-pooled [20:57:56] as it's been used for upgrade trials [21:16:53] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 240 seconds [21:17:02] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 250 seconds [21:17:11] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 260 seconds [21:17:29] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 278 seconds [21:17:29] PROBLEM - MySQL Replication Heartbeat on db1001 is CRITICAL: CRIT replication delay 279 seconds [21:17:38] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 286 seconds [21:17:46] PROBLEM - mysqld processes on db1017 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [21:19:17] RECOVERY - mysqld processes on db1017 is OK: PROCS OK: 1 process with command name mysqld [21:23:29] PROBLEM - MySQL Replication Heartbeat on db1017 is CRITICAL: CRIT replication delay 372 seconds [21:24:04] PROBLEM - MySQL Slave Delay on db1017 is CRITICAL: CRIT replication delay 346 seconds [21:27:12] !log olivneh synchronized php-1.20wmf8/extensions/E3Experiments/experiments/communityClicks.js [21:27:21] Logged the message, Master [21:27:41] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 24 seconds [21:27:41] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 24 seconds [21:27:50] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay 14 seconds [21:27:50] RECOVERY - MySQL Replication Heartbeat on db1017 is OK: OK replication delay 9 seconds [21:28:26] RECOVERY - MySQL Slave Delay on db1017 is OK: OK replication delay 0 seconds [21:28:35] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [21:28:35] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 0 seconds [21:28:44] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 0 seconds [21:31:07] the joys of chained replication [21:32:07] somebody should really fix nagios-wm not to spam like this. [21:32:12] s/fix/rewrite/ [21:33:24] you already excluded yourself from that task iirc [21:33:25] ( [21:33:26] :( [21:33:50] well, i tried [21:33:57] and i sorta kinda did something even [21:34:26] but there are so many things deeply wrong with that short script [21:35:23] at first it wouldnt run for me on python 2.7, then i fixed it and it run, but didnt write anyuthing to irc, then i removed the pokemon exception handling and got errors, then i fixed them, and then it would start to partially work. [21:35:56] of course it wouldnt run on windows, as it uses some crazy file watching features, so i had to comment out that part to develop, and i was unable to test that codepath after i made my changes [21:36:24] so i decided not to bother anyone with the thing i produced, as it was even more half-assed than what is already there. [21:37:04] end rant. [21:41:56] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [21:44:02] PROBLEM - Host db1046 is DOWN: PING CRITICAL - Packet loss = 100% [21:55:01] binasher: heh, I preemptively restarted a background php script...was use 4.3g ram...maybe I should find the leak ;) [21:56:15] I wonder what the fastest way to go about debugging that sort of thing is [21:56:58] AaronSchulz: i'd be curious to see the same script run under 5.4 [21:57:31] does 5.4 have leak fixes (like the closure fix in 5.3.something)? [21:57:44] * AaronSchulz thought it was just better at not wasting as much ram [21:58:26] it has more functional gc supposedly as well [22:02:10] RECOVERY - Host db1046 is UP: PING OK - Packet loss = 0%, RTA = 35.44 ms [22:03:25] AaronSchulz: how is the copy of originals into swift going? [22:04:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [22:05:48] !log aaron synchronized wmf-config/filebackend.php 'Added swift to multiwrite backend for all remaining wikis' [22:05:57] Logged the message, Master [22:06:58] binasher: at this point its just a sync script running frequently [22:07:11] reading from a changelog [22:07:16] PROBLEM - MySQL Replication Heartbeat on db1003 is CRITICAL: CRIT replication delay 246 seconds [22:07:25] PROBLEM - MySQL Slave Delay on db1003 is CRITICAL: CRIT replication delay 255 seconds [22:11:37] RECOVERY - MySQL Replication Heartbeat on db1003 is OK: OK replication delay 0 seconds [22:11:46] RECOVERY - MySQL Slave Delay on db1003 is OK: OK replication delay 0 seconds [22:12:21] AaronSchulz: is that just to keep up with new files after an initial copy completed? [22:12:37] yep [22:13:00] although as of a few minutes ago, writes happen to both synchronously [22:13:35] yay! [22:18:37] binasher: btw, judging from the error logs srv281 is still getting hits sometimes [22:20:42] odd since its commented out in all the pybal conf [22:20:55] i'm not seeing any traffic on port 80 [22:21:01] maybe its from nagios occasionally [22:21:56] Aug 9 22:17:01 10.0.8.31 apache2[17980]: PHP Warning: require(/usr/local/apache/common-local/php-1.20wmf8/includes/WebStart.php) [function.require]: failed to open stream: Permission denied in /usr/local/apache/common-local/php-1.20wmf8/index.php on line 54 [22:22:05] binasher: seems to come in bursts of 2 every ~25min [22:22:09] so that might make sense [22:23:00] actually, every 1 minute or so, I misread that [22:23:24] yep looks like nagios [22:44:55] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [22:59:56] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [23:04:13] can I propose the page move text box be a single line instead of multi lined? [23:06:49] Use bugzilla [23:13:40] Reedy sure, I am just asking if it is worth the effort [23:13:52] Should be a trivial change to make [23:13:57] okay [23:14:25] is this a MediaWiki or Wikimedia thing [23:14:28] ie is it a setting [23:14:56] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [23:15:42] what does rename fall under? [23:17:54] user interface I presume [23:29:56] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [23:41:17] https://bugzilla.wikimedia.org/show_bug.cgi?id=39215