[00:01:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [00:04:04] PROBLEM - SSH on fenari is CRITICAL: Connection refused [00:08:24] !log maxsem synchronized php-1.22wmf10/extensions/GeoData/solrupdate.php [00:08:35] Logged the message, Master [00:10:05] !log maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php [00:10:16] Logged the message, Master [00:15:00] LeslieCarr: I think that we can close https://rt.wikimedia.org/Ticket/Display.html?id=3066 [00:15:07] everything is stopped on spence at this point [00:15:13] it's about to decom [00:15:24] monitoring requires refinement, but ya know [00:15:31] that's a whole different can of worms [00:15:39] New patchset: Dzahn; "re-kill spence from site.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73370 [00:15:40] notpeter: [00:16:09] eh, it's already also in decom [00:16:09] heh [00:16:10] awesome [00:16:18] yeah, but should clean it up from site.pp [00:16:28] I'm sure it's hardcoded all over the place.... [00:16:39] ah, that makes me notice the comment on srv193 [00:16:45] that is also outdated ..now [00:17:06] git grep spence (notpeter) all over [00:17:21] don't grep me, bro ;) [00:17:22] but yeah [00:17:23] oh i did [00:17:25] :~/wmf/puppet/manifests$ git grep spence [00:17:25] decommissioning.pp:'spence', [00:17:27] it should only be in site.pp now [00:17:33] after my change above [00:17:45] Risker, you're getting extra technical if you're joining this channel now :O [00:17:48] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [00:18:18] THO, I just logged in and saw the server lag flag. At least now I know where to get the downlow :) [00:21:43] New patchset: Dzahn; "add comment to srv193 that test has been switched to mw1017" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73371 [00:22:10] :-) [00:22:17] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73371 [00:25:01] !request help Thehelpfulone RT-4880 [00:25:06] :D [00:26:10] I was going to follow up with someone else on that first mutante - I'm not sure whether we necessarily /need/ to mark lists as private - we've not had a problem with it in at least the past couple of years (AFAIK) and it would mean crossposting to lists for announcements etc would be more work [00:26:17] mutante: Might be worth adding something to the srv193 MOTD to say don't use me or something [00:27:06] mutante, can you give me a list n that RT ticket of the lists that are marked as private? [00:27:09] in that* [00:27:45] Thehelpfulone: yea, after moving it to core queue [00:29:12] Reedy: alright, done :) [00:29:22] :) [00:30:10] mutante, sure [00:30:47] Thehelpfulone: done [00:30:57] thanks [00:39:35] Dzahn, Ryan_Lane thanks for test.w.o. I updated https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Trying_tin.27s_code_on_testwiki to mw1017, but don't fully understand that paragraph. [00:42:51] ah. cool. thanks [00:43:17] seems fine to me spagewmf [00:44:18] don't see hardcoded srv193 in /usr/local/bin/ on fenari where the sync scripts are either [00:44:33] mutante/dzahn (?!) I was hoping you could clear the fog around how to rebuild 18n cache on on test.w.o only :) [00:44:36] besidea apache-fast-test which i already fixed [00:46:16] spagewmf: oh.ehm.. afraid not really [00:46:53] but i didnt see that in that section [00:47:30] ah.. yea.. "only way to update these messages is to do a full scap" ..hmm [00:47:48] !log paused updating change_tags indexes on remaining wikis in bug 40867 comment 6 (adds done, drops todo) [00:47:59] Logged the message, Master [00:54:14] mutante, passed it on, I'll let you know (and update the RT ticket) when I have an answer :) [00:54:40] Thehelpfulone: thank you! and cu later then [00:55:02] no problem, when you get a minute perhaps you can !help with RT-2494 ;-) [00:55:26] By help I mean give your comments, I'll do all the hard work! [00:56:44] agree on streamlining it, feel free to remove Jimbo's access , hehehe [00:57:07] i wonder.. does he ever show up on #mediawiki nowadays? [00:58:08] -Notice- {from NickServ} Last seen : Dec 16 00:51:30 2008 (4 years, 29 weeks, 6 days, 01:06:26 ago) [00:58:17] :) [00:58:34] oh correction, that was the wrong nick, but -Notice- {from NickServ} User seen : Dec 01 19:05:06 2012 (31 weeks, 5 days, 06:53:15 ago) [00:59:22] that being said, i can't remember actually needing ops to "deal with trolls" as it says [00:59:39] but looking at it, keeping it updated and possibly removing people not working for us anymore makes sense [01:00:10] one thing you would want ops for is to set the topic, but since we are wiki we should just have mode -t and make it editable [01:00:19] ok, ttyl [01:07:52] spagewmf: (or anyone) you don't happen to roughly know the total number of server the WMF has, do you? [01:08:01] New patchset: Dzahn; "comment another apache module for duplicate definition, add FIXME" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73373 [01:08:07] physical servers, not virtual instances or anything like that [01:26:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.162 second response time [01:38:05] RECOVERY - SSH on fenari is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [01:56:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [01:59:08] Prodego: see http://ganglia.wikimedia.org/latest/ [01:59:16] notice hosts up/hosts down [01:59:30] of course, that's only the systems that are actually in use [01:59:38] we have some that aren't turned on yet [02:16:00] !log LocalisationUpdate completed (1.22wmf9) at Fri Jul 12 02:16:00 UTC 2013 [02:16:11] Logged the message, Master [02:25:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [02:34:20] !log LocalisationUpdate completed (1.22wmf10) at Fri Jul 12 02:34:20 UTC 2013 [02:34:30] Logged the message, Master [02:46:49] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: No successful Puppet run in the last 10 hours [02:47:23] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 12 02:47:23 UTC 2013 [02:47:34] Logged the message, Master [02:52:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [02:56:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [03:23:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [03:28:56] Elsie: bah, you know what I mean [03:50:52] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:54] :-) [03:59:36] Scrollback in here was cute today. Lots of collaboration and updating docs and shit. It was nice to read. [04:01:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:52] PROBLEM - Puppet freshness on db78 is CRITICAL: No successful Puppet run in the last 10 hours [04:03:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [04:11:10] Elsie: :P [04:22:57] New patchset: Ori.livneh; "EventLogging module: tabs -> spaces; no significant changes." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73387 [04:32:42] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:39:42] PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on mw1007 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on mw121 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:42] PROBLEM - Puppet freshness on mw1197 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:43] PROBLEM - Puppet freshness on mw1041 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:43] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:44] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:44] PROBLEM - Puppet freshness on solr1003 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:45] PROBLEM - Puppet freshness on search1024 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:45] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:46] PROBLEM - Puppet freshness on sq76 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:46] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:47] PROBLEM - Puppet freshness on search18 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:47] PROBLEM - Puppet freshness on solr3 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:48] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:48] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [04:40:49] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:42] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:43] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [04:41:43] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:44] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:44] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:45] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:45] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:46] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:46] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:47] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:47] PROBLEM - Puppet freshness on pc1 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:48] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 10 hours [04:41:48] PROBLEM - Puppet freshness on praseodymium is CRITICAL: No successful Puppet run in the last 10 hours [04:41:49] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:49] PROBLEM - Puppet freshness on rdb1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:50] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:50] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:51] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [04:41:51] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on mw1189 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:42] PROBLEM - Puppet freshness on mw1069 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:43] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:43] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:44] PROBLEM - Puppet freshness on mw1150 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:44] PROBLEM - Puppet freshness on mw1205 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:45] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:45] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:46] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:46] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:47] PROBLEM - Puppet freshness on mw79 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:47] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:48] PROBLEM - Puppet freshness on search33 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:48] PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:49] PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours [04:42:49] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:42] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:42] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:42] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [04:43:42] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:42] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:42] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [04:45:42] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:42] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:42] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:42] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:43] PROBLEM - Puppet freshness on db1014 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:43] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:44] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:44] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:45] PROBLEM - Puppet freshness on mc1012 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:45] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:46] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:46] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:47] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:47] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:48] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:48] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:49] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:49] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:50] PROBLEM - Puppet freshness on solr1 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:50] PROBLEM - Puppet freshness on search25 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:51] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:51] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:52] PROBLEM - Puppet freshness on virt11 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:52] PROBLEM - Puppet freshness on wtp1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:46:42] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:46:42] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:46:42] PROBLEM - Puppet freshness on capella is CRITICAL: No successful Puppet run in the last 10 hours [04:46:42] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:46:42] PROBLEM - Puppet freshness on cp1017 is CRITICAL: No successful Puppet run in the last 10 hours [04:48:42] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [04:48:42] PROBLEM - Puppet freshness on mw1021 is CRITICAL: No successful Puppet run in the last 10 hours [04:48:42] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [04:48:42] PROBLEM - Puppet freshness on db77 is CRITICAL: No successful Puppet run in the last 10 hours [04:48:42] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:42] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:43] PROBLEM - Puppet freshness on db51 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:43] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:44] PROBLEM - Puppet freshness on es5 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:44] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:45] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:45] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:46] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:46] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:47] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:47] PROBLEM - Puppet freshness on sq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:48] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:49:48] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [04:49:49] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:42] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:43] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:43] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:44] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:44] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:45] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:45] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:46] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:46] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:47] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:47] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:48] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:48] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:49] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:49] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:50] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:50] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on db31 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:42] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:43] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:43] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:44] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:44] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:45] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:45] PROBLEM - Puppet freshness on search1006 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:46] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [04:51:46] PROBLEM - Puppet freshness on search1011 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:47] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:47] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [04:51:48] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [04:52:42] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [04:52:42] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:52:42] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [04:52:42] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [04:52:42] PROBLEM - Puppet freshness on db59 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:42] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:42] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:42] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:42] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:42] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:42] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:42] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:42] PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:42] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:42] PROBLEM - Puppet freshness on cp1068 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:42] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:42] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:42] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:42] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:42] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:43] PROBLEM - Puppet freshness on db1028 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:43] PROBLEM - Puppet freshness on db38 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:44] PROBLEM - Puppet freshness on db68 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:44] PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours [04:56:45] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:45] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:46] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:46] PROBLEM - Puppet freshness on mw1039 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:47] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:47] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:48] PROBLEM - Puppet freshness on mw120 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:48] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:49] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:49] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:42] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:42] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:42] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:42] PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:42] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:42] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:42] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:42] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:42] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:42] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:11] New review: Ori.livneh; "Thanks for the review. Replies in-line; updated patch forthcoming." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71927 [05:00:42] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:42] PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:42] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:42] PROBLEM - Puppet freshness on db1053 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:42] PROBLEM - Puppet freshness on mw102 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:42] PROBLEM - Puppet freshness on es8 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:42] PROBLEM - Puppet freshness on mw6 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:43] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:43] PROBLEM - Puppet freshness on srv241 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:44] PROBLEM - Puppet freshness on virt7 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:44] PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:45] PROBLEM - Puppet freshness on ms1002 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:45] PROBLEM - Puppet freshness on wtp1023 is CRITICAL: No successful Puppet run in the last 10 hours [05:01:42] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [05:01:42] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [05:01:42] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [05:01:42] PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours [05:01:42] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:42] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:42] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:42] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:42] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:42] PROBLEM - Puppet freshness on cp1045 is CRITICAL: No successful Puppet run in the last 10 hours [05:04:42] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [05:04:42] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [05:04:42] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [05:04:42] PROBLEM - Puppet freshness on db52 is CRITICAL: No successful Puppet run in the last 10 hours [05:04:42] PROBLEM - Puppet freshness on mc1003 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:42] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:42] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:42] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:42] PROBLEM - Puppet freshness on cp1061 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:42] PROBLEM - Puppet freshness on cp3007 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:12] New patchset: Ori.livneh; "EventLogging module: tabs -> spaces; no significant changes." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73387 [05:06:12] New patchset: Ori.livneh; "Rewrite of EventLogging module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73388 [05:07:00] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:00] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:00] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:00] PROBLEM - Puppet freshness on cp1003 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:00] PROBLEM - Puppet freshness on cp1051 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:32] oh, god damn it [05:07:40] i must have deleted the change-id line when amending the commit [05:08:33] Is it just me or does icinga-wm voice itself like a million times? [05:09:00] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:00] PROBLEM - Puppet freshness on lvs1 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:00] PROBLEM - Puppet freshness on mc1009 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:00] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:00] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:00] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:00] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:01] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:01] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:02] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:02] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:03] PROBLEM - Puppet freshness on mw1214 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:03] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:04] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:04] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:05] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:05] PROBLEM - Puppet freshness on solr2 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:06] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:06] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:07] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:07] PROBLEM - Puppet freshness on wtp1018 is CRITICAL: No successful Puppet run in the last 10 hours [05:09:47] the channel is configured to auto-voice it on join, and it's flooding itself off and re-joining in a loop [05:10:00] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:00] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:00] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:00] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:00] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:06] I only see it voicing, no parts or joins. [05:10:14] * Jasper_Deng thinks Elsie has conference mode on [05:10:18] I see the quits and joins [05:10:21] 22:10 icinga-wm has left IRC (Excess Flood) [05:10:21] 22:10 I only see it voicing, no parts or joins. [05:10:27] I don't think irssi has conference mode. [05:10:30] Oh. [05:10:35] I have it ignored. [05:10:38] Heh. [05:10:44]