[00:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 00:02:37 UTC 2013 [00:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [00:05:38] twkozlowski: my unederstanding is the default was changed back for all. I think the patch was by Krinkle? [00:05:48] understanding* [00:05:52] I'll do some hunting. [00:06:54] I don't recall changing wgCookieExpiration in either core of wmf. [00:06:58] but its possible [00:07:23] core: $wgCookieExpiration = 180 * 86400; [00:07:41] New patchset: Ori.livneh; "Clean-up of 'eventlogging::archive' class." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74562 [00:07:42] wmf: $wgCookieExpiration = 30 * 86400; [00:07:48] StevenW: twkozlowski [00:07:58] (latest master) [00:08:20] I may be confusing it was something related but different. [00:09:03] That patch about removing the anon cookie that expired waaay too long? [00:09:44] LeslieCarr: do you happen to know where the pybal config file for the squids live? [00:09:59] mwalker: fenari [00:10:07] like everything else ;) [00:10:26] ah; cool; where on that beast? [00:10:37] /home/w/conf/pybal [00:10:50] *thumbs up* [00:10:54] thanks! [00:10:57] yw [00:11:07] https://www.mediawiki.org/w/index.php?title=Manual:%24wgCookieExpiration&action=purge [00:14:35] thanks Krinkle [00:15:03] StevenW: Yes, but you responded to Raymond's question about that on the privacy policy discussion page on Meta :-) [00:15:45] I mean, the guy's question was about the Wikimedia set up, not the general MediaWiki login expiration. So I think the answer still applies [00:16:48] "This year we noted the discrepancy in engineering and reset $wgCookieExpiration back to 30 days, and I've updated the MediaWiki manual page you linked to note this." [00:18:05] https://www.mediawiki.org/w/index.php?title=Manual:$wgCookieExpiration&diff=724468&oldid=723903 [00:18:17] so Krinkle says it actually wasn't, that's why I pinged you StevenW :) [00:23:30] twkozlowski: The commit date doesn't resemble deployment for the svn commit (probably several months later), but yes, as I see it, I'd say there was about a year where the expiration was 180 days. [00:23:43] However thinking back I find that unlikely, afaik we've always had 30 days in production. [00:24:07] perhaps it was reset to 30 by other means, or maybe it wasn't deployed as-is. [00:24:16] in svn days we had quite a branch mess [00:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 00:24:52 UTC 2013 [00:25:02] Krinkle: certainly we didn't always have 30 days in production [00:25:22] People have been complaining about the change from 180 to 30, and I myself remember we used to have 180 [00:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [00:27:49] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 00:27:45 UTC 2013 [00:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [00:29:16] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 00:29:07 UTC 2013 [00:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [00:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 00:32:41 UTC 2013 [00:33:07] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [00:34:37] is there a good pattern for composing a single configuration file out of multiple resources? i don't like the rsync module's approach of creating a directory of file fragments and combine them at the end of the run; it seems weird to use /etc as storage medium for intermediate results [00:34:47] ori-l: nope [00:34:50] twkozlowski: I don't know if we ever had 180 in production, but looking purely at the commits I'd say that does indeed seem possible. However without other proof I wouldn't be so sure. [00:34:52] twkozlowski: I guess you have more data? [00:35:04] ori-l: that's something that puppet basically can't handle [00:35:16] hence the hack for rsync and puppet (and probably a bunch of other things) [00:35:17] it's really incredible the sort of things you can't do or can't do well with puppet [00:35:22] yep [00:35:23] Krinkle: Just my memory and a talk page post by a different user [00:35:32] twkozlowski: ok [00:35:33] ori-l: if there was iteration it would be doable [00:35:43] but alas, none in puppet [00:36:25] what about using the fact that arrays and hashes are mutable? [00:37:05] you could update a global var from multiple places, and use stages / virtual resources / some other sequencing trick to ensure that the variable's value is queried and the config file generated at the end [00:37:08] I guess it's possible to put some chaining in and add to arrays and hashes [00:37:12] then at the end make the file [00:37:22] but then you need to deal with chaining [00:37:29] or stages (which are totally fucking broken in puppet) [00:37:53] i like that puppet labs is at least fairly candid and apologetic of the basic design flaws, of which there are many [00:38:11] also, I'm not sure if you can actually modify a global variable that way [00:38:20] it seems like every third documentation page has a box with a note like: [00:38:28] "we botched this horribly; you probably shouldn't use this." [00:38:41] unfortunately a lot of those were best practices at some point [00:38:51] yes. [00:38:54] parametrize all classes! [00:38:55] wait, don't! [00:38:59] :D [00:40:02] it really seems like that realize they fucked up and they rush in some hack to fix it [00:40:15] and later realize "OH NO, this is horrible too!" [00:40:18] yes, and always by introducing a sweeping, new abstraction [00:40:29] @seen bblack [00:40:46] of course, I'm fighting puppet right now and I'm pissed off at it, so maybe I'm being overly harsh [00:41:37] here's what just happened to me: a module was broken, so it didn't compile it and used an old cached copy [00:41:58] I restart the puppet master, it shows me the error, I run puppet again, it shows me a different error [00:42:02] and caches the broken module [00:42:26] heh [00:42:26] so much fail [00:42:41] Wait, so if there's a parse error it's just, like, hey, I'll try this instead! [00:43:11] andrewbogott: it did for me [00:43:21] not with a parse error, i don't think [00:43:24] That makes it /very/ hard to debug! [00:43:36] ori-l: mine was a parse error :( [00:43:42] but with other errors it could get stuck in different places because the order of execution is non-determinsitic, i think [00:43:50] oh, heh, wonderful. [00:43:53] Although I guess it fits with the "This is terrible but seems to work for now" design ethos [00:45:52] i think the ease with which you can extend puppet with custom resource types, providers and ruby functions is one thing they got profoundly right; you can bend it to create concise DSLs for managing the configuration of complex software [00:46:07] i really enjoy that part of it [00:46:13] yeah, that's nice [00:46:50] but why param classes an definitions? [00:46:54] *and [00:47:00] what's the major difference? [00:47:33] there are many things in puppet which are roughly equivalent and where the salient difference boils down to 'broken in different ways' [00:47:36] including and using functions is a pretty massive pain in the ass [00:47:42] though I guess it's easier in modules [00:47:57] it's also discouraged [00:48:19] yeah, but i don't take puppet labs' pronouncements on these things as binding [00:48:23] custom facter is also a pain [00:48:33] because you have to ensure it's deployed before it's used [00:48:33] because unfortunately a lot of those were best practices at some point [00:49:17] what's custom facter? [00:49:21] some external tool that generates facts? [00:49:35] it's just adding a ruby facter script [00:49:47] that adds some fact [00:49:47] that you then want to use in puppet [00:49:49] but you deploy it from puppet [00:50:03] ah, vagrant has a nice hook for that that i've used; never tried implementing it from scratch [00:50:22] New patchset: TTO; "(bug 51605) add alias for project namespace on ckbwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74567 [00:50:36] this is one thing that salt handles amazingly well. you have grains, modules, pillars, etc on the master and it'll fetch it before its used [00:51:06] grain = fact, pillar = global variable, module = custom function [00:51:15] state = class [00:51:42] hmm, that sounds pretty cool [00:51:43] you can use custom modules in states and it'll ensure it's installed before use. [00:52:04] and you can also use those modules for remote execution [00:52:14] you can also call individual states via remote execution [00:52:53] salt's states are deficient in functionality compared to puppet, though, so it's a give and take [00:54:22] ori-l: hm. is that hook used in vagrant common in puppet too? [00:54:30] I'm not sure how we're currently doing it, but it sucks [00:54:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 00:54:41 UTC 2013 [00:55:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [00:56:33] i think vagrant just assembles all custom facts into FACTER_%s environment variables and it uses the shell trick of defining a variable for a single command by including its definition as a prefix of the invocation [00:57:01] like IFS='|' awk ... [00:57:25] ah [00:57:51] ok. time to go climbing [00:58:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 00:58:48 UTC 2013 [00:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 01:02:42 UTC 2013 [01:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [01:14:47] New patchset: TTO; "(bug 51537) require 10 edits for autoconfirmed on ckbwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74568 [01:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 01:24:47 UTC 2013 [01:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [01:26:42] gitblit seems to be unresponsive: https://git.wikimedia.org/tree/mediawiki%2Fcore [01:27:44] New patchset: Vogone; "Enabling the 'property-create' right for all users on testwikidatawiki Bug: 51637" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74528 [01:27:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 01:27:39 UTC 2013 [01:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [01:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 01:28:50 UTC 2013 [01:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [01:32:56] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 01:32:46 UTC 2013 [01:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [01:39:14] hello operations [01:39:14] can I ask something ? [01:40:52] I'd like to pitch an idea [01:41:34] but probably not the best time since it's late [01:42:20] I'd like to pitch the idea of having skeleton git repositories for different projects that require packaging [01:42:59] so for example, there would be a Python skeleton , a Ruby skeleton, a Java skeleton, a C or C++ skeleton , a Perl skeleton and a PHP skeleton [01:43:04] each in different repos [01:43:27] and the aim would be, that if someone needs to do a debianization, he could clone on of those skeletons, and fit his applications in it, and push [01:43:47] and in doing this, he would have more chances of doing it the right way [01:44:28] it's the opposite of what we're doing now(at least in terms of order) [01:44:43] right now we're developing and considering debianization as the last step [01:45:36] but if we had a way to use somethng that operations provided from the get-go, which they agree on, then we'd be able to build on top of that and then the whole process would be easier IMHO [01:54:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 01:54:42 UTC 2013 [01:55:42] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [01:58:06] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 01:58:04 UTC 2013 [01:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [01:58:46] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 01:58:45 UTC 2013 [01:59:17] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [02:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 02:02:39 UTC 2013 [02:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:30] !log LocalisationUpdate completed (1.22wmf10) at Fri Jul 19 02:15:29 UTC 2013 [02:15:41] Logged the message, Master [02:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 02:24:50 UTC 2013 [02:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [02:27:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 02:27:35 UTC 2013 [02:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 02:28:47 UTC 2013 [02:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:56] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [02:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 02:32:43 UTC 2013 [02:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [02:33:48] !log LocalisationUpdate completed (1.22wmf11) at Fri Jul 19 02:33:48 UTC 2013 [02:33:59] Logged the message, Master [02:49:50] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 19 02:49:50 UTC 2013 [02:50:01] Logged the message, Master [02:56:04] New review: Hazard-SJ; "Shouldn't that be after test2wiki? That's my only issue." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/74528 [02:57:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 02:57:34 UTC 2013 [02:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [02:58:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 02:58:46 UTC 2013 [02:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [03:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 03:02:37 UTC 2013 [03:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [03:03:54] New review: Vogone; "If you want to keep an alphabetical order, test2wiki would also have to be before testwiki." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/74528 [03:13:32] New review: Hazard-SJ; "Since the others are also misplaced, might as well a separate patch be created to fix that, so this ..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/74528 [03:24:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 03:24:45 UTC 2013 [03:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [03:27:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 03:27:40 UTC 2013 [03:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [03:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 03:28:46 UTC 2013 [03:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [03:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 03:32:44 UTC 2013 [03:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [03:39:16] https://git.wikimedia.org appears to be down. [03:54:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 03:54:44 UTC 2013 [03:55:40] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:16] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 03:58:08 UTC 2013 [03:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 03:58:49 UTC 2013 [03:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [04:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 04:02:43 UTC 2013 [04:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:46] PROBLEM - Disk space on ms-be1 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sdh1 is not accessible: Input/output error [04:24:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 04:24:45 UTC 2013 [04:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [04:27:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 04:27:37 UTC 2013 [04:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [04:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 04:28:49 UTC 2013 [04:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [04:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 04:32:40 UTC 2013 [04:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 04:54:44 UTC 2013 [04:55:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 04:57:38 UTC 2013 [04:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [04:58:46] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 04:58:44 UTC 2013 [04:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [05:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 05:02:38 UTC 2013 [05:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [05:24:46] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 05:24:44 UTC 2013 [05:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [05:27:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 05:27:38 UTC 2013 [05:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [05:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 05:28:50 UTC 2013 [05:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [05:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 05:32:40 UTC 2013 [05:32:56] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [05:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [05:44:56] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [05:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 05:54:47 UTC 2013 [05:55:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [05:57:46] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 05:57:40 UTC 2013 [05:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [05:58:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 05:58:46 UTC 2013 [05:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [06:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 06:02:38 UTC 2013 [06:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [06:13:26] PROBLEM - HTTP radosgw on ms-fe1004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.004 second response time [06:15:26] PROBLEM - HTTP radosgw on ms-fe1003 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.003 second response time [06:15:42] oh ceph [06:16:06] PROBLEM - HTTP radosgw on ms-fe1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.014 second response time [06:19:17] grrr [06:20:33] ceph? [06:21:49] yeah [06:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 06:24:50 UTC 2013 [06:25:37] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:06] RECOVERY - HTTP radosgw on ms-fe1002 is OK: HTTP OK: HTTP/1.1 200 OK - 311 bytes in 0.003 second response time [06:27:47] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 06:27:41 UTC 2013 [06:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 06:28:48 UTC 2013 [06:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 06:32:41 UTC 2013 [06:32:56] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [06:33:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [06:36:41] New patchset: Ori.livneh; "Clean-up of 'eventlogging::archive' class." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74562 [06:48:40] still up ori-l? [06:48:54] yeah; what's up? [06:49:00] no just saying [06:49:06] since i saw your commit :) [06:50:11] ah, yeah, i get a bit fanatical sometimes about a set of changes. [06:50:20] "sometimes" [06:50:33] New review: Faidon; "Unparameterizing it and hardcoding all_networks inside EL seems like a step in the opposite directio..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/74562 [06:50:39] er, s/opposite/wrong/ even [06:50:58] Aaron|home: past your bed-time, too :P [06:51:11] * Aaron|home has laundry still spinning [06:51:35] of course that's just an excuse since I'm almost always up at this time [06:52:09] i took you to be speaking euphemistically anyhow ;) [06:52:26] * Aaron|home set his s4 lock from pin -> password -> pin again since the typo rate was too high [06:52:39] and all the letters cover up my Liz Taylor wallpaper [06:53:03] S4? [06:53:23] samsung galazy [06:54:04] the number keypad fits just right though [06:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 06:54:51 UTC 2013 [06:55:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [06:57:36] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 06:57:35 UTC 2013 [06:58:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [06:58:46] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 06:58:41 UTC 2013 [06:58:56] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [06:58:57] New review: Ori.livneh; "Because the module does not otherwise employ parametrized classes or rely on the ordering of resourc..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74562 [06:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [06:59:39] knight to d6, faidon! :P [07:00:28] how's ordering related? [07:00:54] you mean that you can include ::archive from multiple places [07:01:22] right, but the resource declaration style of loading the class would then have to load first [07:02:05] so IIRC it's possible (and happens in our codebase) that you have class { 'eventlogging::archive': param => 'foo', } in one place, and 'include eventlogging::archive' in others [07:02:36] which is OK unless some other dependency causes puppet to instantiate the class on the include statement before it acts on the resource declaration style instantiation [07:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 07:02:40 UTC 2013 [07:02:56] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:23] it sounds obscure, but i come across it regularly. it bit me when i made the changes to the scap scripts. [07:03:38] why would you include it in one place and include it parameterized in another? [07:03:56] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [07:03:59] hi paravoid [07:04:16] good morning [07:04:58] good morning [07:05:21] you might include it in eventlogging::consumer, reasoning that any node that consumes events ought to have log rotation for /var/log/eventlogging, but you don't know in the resource definition about what rsync hosts will pull the data [07:05:28] paravoid: pushed some new patches in => https://gerrit.wikimedia.org/r/#/c/73860/ [07:05:54] paravoid: I addressed your comments and Brandon's [07:06:12] poor paravoid :) [07:06:21] average: you grab the left arm, i'll grab the right arm [07:07:00] hahaha [07:07:10] :D [07:07:17] average: will you be around in 20'? [07:07:26] I need to finish up with something else [07:07:31] paravoid: I have insomnia, I'll be around in * [07:07:38] *' [07:07:38] and Ori, and it's late for him too :) [07:07:52] but you're in Europe aren't you? [07:07:59] I am in Europe yes [07:08:12] ok, I got confused with the insomnia comment :) [07:08:25] my comments? huh? [07:08:35] jorm: brandon black [07:08:40] your evil alter-ego [07:08:53] hehe [07:08:54] the False Brandon. [07:09:25] he says the same about you [07:09:31] told us not to believe a word you said [07:09:45] ori-l: hm, logrotate and rsync at the same place feels weird [07:09:55] maybe that's the root cause of your issue [07:10:18] maybe rsync should move to the role? [07:10:28] I WAS THE FIRST. [07:10:46] jorm: https://en.wikipedia.org/wiki/Genetic_fallacy [07:10:49] maybe, yeah [07:10:57] or a different class, parameterized [07:11:19] there's just something nice about having this entire module completely dodge an entire class of puppet pitfalls (duplicate class declarations), and it somehow seems unfair to compromise that over such a small detail as rsync hosts_allowed [07:11:34] so role class it is [07:13:05] lol [07:13:19] I don't think we need to fear parameterized classes that much [07:13:34] your rsync accessor class could fail if you don't pass its parameters, if you prefer that [07:13:52] (if ... { fail("You need to supply a parameter to ...") } [07:15:11] i just flipped through this o'reilly book, "puppet types and providers" [07:15:27] and like a man-child with a new toy i keep looking for an excuse to implement a type or a provider in ruby [07:15:45] and it's taking every ounce of self-restraint that i don't have not to invent some new abstractions [07:15:59] so i'm just going to move it to the role class and call it a night :P [07:17:04] hahaha [07:24:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 07:24:51 UTC 2013 [07:25:26] RECOVERY - HTTP radosgw on ms-fe1004 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 186 bytes in 0.001 second response time [07:25:26] RECOVERY - HTTP radosgw on ms-fe1003 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 186 bytes in 0.011 second response time [07:25:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [07:26:02] oh, ceph.... [07:28:16] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 07:28:07 UTC 2013 [07:28:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [07:28:56] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 07:28:52 UTC 2013 [07:29:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [07:33:06] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 07:33:01 UTC 2013 [07:34:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [07:35:17] this is me sleeping btw, but now I'm not :) http://garage-coding.com/sleeping.png [07:35:56] that was random [07:38:56] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [07:38:56] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:38:56] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:38:56] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:38:56] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [07:38:56] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:38:56] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [07:43:06] New patchset: Ori.livneh; "Remove 'eventlogging::archive' class." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74562 [07:46:53] Aaron|home: still around? [07:47:34] * Aaron|home is not around [07:48:16] I just want to show you something [07:48:49] Aaron|home: http://tracker.ceph.com/issues/5675 [07:48:52] epic bug is epic [07:50:40] :/ [07:52:22] also http://tracker.ceph.com/issues/5674 [07:52:24] and http://tracker.ceph.com/issues/5655 [07:52:33] and 5671 but this was a misunderstaing [07:52:37] misunderstanding [07:52:46] but the peering bug seems to be fixed, so at least there are some good news [07:53:37] and they wanted to release an rc today [07:53:39] bwahahaha [07:54:47] okay, stopping all radosgws [07:54:50] no point really [07:54:56] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Jul 19 07:54:46 UTC 2013 [07:55:25] !log stopping radosgw across the ceph cluster, post-upgrade issues [07:55:35] Logged the message, Master [07:55:36] PROBLEM - Puppet freshness on cp1042 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:06] PROBLEM - HTTP radosgw on ms-fe1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.001 second response time [07:57:24] it gets the semantics of HTTP 500 exactly right [07:57:26] PROBLEM - HTTP radosgw on ms-fe1003 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.003 second response time [07:57:36] PROBLEM - HTTP radosgw on ms-fe1001 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 758 bytes in 0.002 second response time [07:58:06] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Jul 19 07:58:03 UTC 2013 [07:58:26] PROBLEM - Puppet freshness on cp1044 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:46] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Jul 19 07:58:43 UTC 2013 [07:58:47] okay, reviewing mode now [07:59:16] PROBLEM - Puppet freshness on cp1041 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:17] oh niiice [07:59:28] thank you [08:02:46] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Jul 19 08:02:39 UTC 2013 [08:03:06] PROBLEM - Puppet freshness on cp1043 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:25] New review: Faidon; "Perfect, thank you!" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/74562 [08:03:26] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/74562 [08:04:24] weeee, thanks [08:06:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:05] average: hey [08:07:07] 26 [08:07:07] 27 [08:07:07] dclass_java.h: 28 [08:07:07] » » javac -classpath ./java java/dclass/dClass.java 29 [08:07:09] » » javah -o java/jni/dclass_java.h -classpath ./java dclass.dClass [08:07:13] what does that do? [08:07:27] (first of all, one tab is enough, but let's ignore that for a moment) [08:07:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [08:07:57] creating libdclass_java.h out of dClass.java? [08:08:01]