[00:00:43] (03CR) 10Reedy: [C: 032] Use uca-cy collation on Welsh projects [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106163 (owner: 10Reedy) [00:00:52] (03Merged) 10jenkins-bot: Use uca-cy collation on Welsh projects [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106163 (owner: 10Reedy) [00:01:58] !log reedy synchronized wmf-config/InitialiseSettings.php [00:02:05] Logged the message, Master [00:16:16] (03PS1) 10BryanDavis: Ensure that logstash is running [operations/puppet] - 10https://gerrit.wikimedia.org/r/106164 [00:27:36] (03CR) 10Ori.livneh: [C: 032] "Keep in mind that having the service be automatically refreshed (read: restarted) by Puppet may not be desirable in the long run, if peopl" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106164 (owner: 10BryanDavis) [00:35:17] !log kaldari synchronized php-1.23wmf9/extensions/VectorBeta [00:35:23] Logged the message, Master [00:36:28] (03PS1) 10BryanDavis: Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/106169 [00:36:29] (03PS1) 10BryanDavis: Proxy kibana.wikimedia.org via misc varnish cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 [00:37:47] (03Abandoned) 10BryanDavis: Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis) [00:41:37] !log reedy synchronized php-1.23wmf9/extensions/CategoryTree/CategoryTreeFunctions.php [00:41:44] Logged the message, Master [00:42:02] (03PS2) 10BryanDavis: Add kibana.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/105105 [00:43:17] !log reedy synchronized php-1.23wmf9/resources 'touch' [00:43:24] Logged the message, Master [00:53:46] I'm trying to do a git push to gerrit, but it says my authentication fails. I'm using the same username and password as I do for the gerrit website. Is there something special I need to do when doing git push from the command line? [00:54:00] sorry, wrong channel [01:01:27] !log reedy updated /a/common to {{Gerrit|Ic65d167b3}}: Use uca-cy collation on Welsh projects [01:01:31] (03PS1) 10Reedy: Remove $wgCategoryTreeDisableCache, leave as default of true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106172 [01:01:33] Logged the message, Master [01:04:07] (03CR) 10Reedy: [C: 032] Remove $wgCategoryTreeDisableCache, leave as default of true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106172 (owner: 10Reedy) [01:04:17] (03Merged) 10jenkins-bot: Remove $wgCategoryTreeDisableCache, leave as default of true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106172 (owner: 10Reedy) [01:04:54] !log reedy synchronized wmf-config/ [01:05:01] Logged the message, Master [01:05:11] (03PS2) 10BryanDavis: [WIP] Add logstash config for udp2log [operations/puppet] - 10https://gerrit.wikimedia.org/r/106154 [01:06:06] !log reedy synchronized php-1.23wmf9/extensions/CategoryTree/CategoryTreeFunctions.php 'rv' [01:06:12] Logged the message, Master [01:06:54] !log reedy updated /a/common to {{Gerrit|I613b194f9}}: Remove $wgCategoryTreeDisableCache, leave as default of true [01:07:00] Logged the message, Master [01:07:03] (03PS1) 10Reedy: Fix indenting in $wmgUseCategoryTree [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106174 [01:07:18] (03CR) 10Reedy: [C: 032] Fix indenting in $wmgUseCategoryTree [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106174 (owner: 10Reedy) [01:07:27] (03Merged) 10jenkins-bot: Fix indenting in $wmgUseCategoryTree [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106174 (owner: 10Reedy) [01:14:33] !log reedy synchronized wmf-config/ [01:14:41] Logged the message, Master [01:17:33] !log reedy updated /a/common to {{Gerrit|Ib7ed56216}}: Fix indenting in $wmgUseCategoryTree [01:17:37] (03PS1) 10Reedy: Remove wmgCategoryTreeDynamicTag [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106180 [01:17:40] Logged the message, Master [01:18:06] !log kaldari synchronized php-1.23wmf9/extensions/MobileFrontend/ [01:18:12] Logged the message, Master [01:18:38] (03CR) 10Reedy: [C: 032] Remove wmgCategoryTreeDynamicTag [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106180 (owner: 10Reedy) [01:18:59] (03Merged) 10jenkins-bot: Remove wmgCategoryTreeDynamicTag [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106180 (owner: 10Reedy) [01:19:41] Reedy: can i pop into test wiki, mw1017, i want to remove a single try/catch block so we get the original exception in the errors logs (mediawiki throws away information about nested exceptions). I can't get the error to reproduce on our labs or in vagrant (or in beta) [01:20:01] Reedy: in the flow exception [01:20:05] s/exception/extension/ [01:20:35] !log reedy synchronized wmf-config/ [01:20:42] Logged the message, Master [01:25:46] !log bsitu synchronized php-1.23wmf9/extensions/Flow 'Revert "Utilize BufferedCache in TreeRepository"' [01:25:52] Logged the message, Master [02:02:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 11:01:51 PM UTC [02:20:11] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:21] PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:28] !log LocalisationUpdate completed (1.23wmf9) at Wed Jan 8 02:27:28 UTC 2014 [02:50:40] Is wikitech down? [02:51:55] !log LocalisationUpdate completed (1.23wmf8) at Wed Jan 8 02:51:54 UTC 2014 [03:01:01] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.252 second response time [03:01:11] RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.548 second response time [03:06:03] (Now appears to be back up.) [03:26:40] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jan 8 03:26:40 UTC 2014 [03:26:46] Logged the message, Master [03:27:24] James_F: again? [03:27:34] huh [03:27:40] 21:25 <+icinga-wm> PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:47] virt0 == wikitech [03:27:55] Yeah. It came back. [03:27:59] yeah [03:28:11] just, confirming that a bot saw the same as you :) [05:03:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 11:01:51 PM UTC [06:43:38] (03CR) 10Jeremyb: [C: 031] "This fixes some broken redirects in addition to the refactoring/cleanup." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106107 (owner: 10Jeremyb) [08:04:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 11:01:51 PM UTC [08:04:33] (03PS9) 10Physikerwelt: Add Mathoid module (TeX -> MathML / SVG conversion web service) [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 [08:05:03] (03CR) 10Physikerwelt: "@GWicke: Any updates here?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 (owner: 10Physikerwelt) [09:08:28] (03PS1) 10Yuvipanda: Deploy Extension:MobileApp to betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 [09:08:40] hashar: ^ is all that is needed in this case, I think? [09:08:48] I'm creating a tracking bug now [09:09:56] YuviPanda: and probably need an entry in extension-list-labs [09:10:07] though you don't have any i18n message [09:10:14] hashar: hmm, extension-list-labs is empty [09:10:17] is that expected? [09:10:18] ahh [09:10:34] yeah it get emptied whenever extensions are moved to production [09:10:41] in that case, it lands in extension-list [09:10:44] ah [09:10:45] ok [09:10:48] i'll just add this there then [09:10:50] as in [09:10:55] to extension-list-labs [09:11:00] but I think we have some $wg setting for it nowadays [09:11:33] (03PS2) 10Yuvipanda: Deploy Extension:MobileApp to betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 [09:11:38] hmm, updated it anyway [09:12:47] hashar: it might still be used, I see a wgExtensionEntryPointListFiles in CommonSettings-labs.php [09:14:44] YuviPanda: yeah that variable :D [09:14:50] :) [09:14:51] can't remember how it works though [09:14:54] heh [09:14:58] i think it is used by mergeMessages.php script [09:15:01] but added the extension to that list anyway [09:15:12] so you might not have to use extension-list-labs [09:15:18] and can amend the wiki doc :-D [09:15:53] hashar: well, $wgExtensionEntryPointListFiles[] = "$wmfConfigDir/extension-list-labs"; [09:15:59] hashar: so that file is probably still being used :D [09:17:23] hashar: and I guess the doc is accurate then [09:18:44] hashar: think you can merge that patch? :D [09:19:39] have you filled form 804x3 and got it signed by your chain of commandment? [09:20:17] I'm looking for a pigeon to attach the three copies of 804 for transport [09:20:23] the gargoyles seem to keep eating them all [09:20:31] :D [09:20:43] :P [09:20:59] hashar: do you want me to get approval from someone before merging this? [09:21:07] hashar: greg-g perhaps? or someone else? [09:21:27] na I don't give a shit about approvals [09:21:36] or we will end up a giant bureaucratic organization [09:21:55] I am not sure why we need another extension to push a .less file but I guess you have a valid use case :-D [09:22:01] WIN! :) [09:22:46] hashar: <3 :) [09:24:35] hashar: use case being that they are two very different products, and if I put these files in MobileFrontend then the team that maintains it will never really be using it, and then it will probably just rot [09:24:43] or I'll have to step into using MF properly [09:24:55] hashar: MF also has a slightly complicated RL setup, because of targets and such [09:25:03] which the app does not need at all [09:25:46] hashar: two teams, so separate extensions make sense? [09:26:03] (03PS3) 10Hashar: Deploy Extension:MobileApp to betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 (owner: 10Yuvipanda) [09:27:34] (03CR) 10Hashar: [C: 04-1] "The change itself is good but can't be deployed because mediawiki/extensions.git is currently broken. That prevents MobileApp code to be d" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106217 (owner: 10Yuvipanda) [09:27:40] YuviPanda: so yeah that would works [09:27:41] BUT [09:27:58] code is deployed using mediawiki/extensions.git which has been broken for a few days now [09:28:26] so if we merge that change in, MobileApp/MobileApp.php does not exist and will cause beta to dies :/ [09:28:47] gotta wait for mediawiki/extensions.git to be fixed up which is https://bugzilla.wikimedia.org/show_bug.cgi?id=49846 [09:29:07] hashar: ah, right. [09:29:28] hashar: is anyone looking at it? [09:29:41] pinged Chad about it yesterday, but apparently it is not fixed [09:29:50] and I have no clue what needs to be done [09:30:04] pfft, sigh [09:30:05] ok [09:30:10] sorry :( [09:30:20] hashar: nah, that's fine :) [09:30:56] hashar: I'm planning ahead so we don't run into random last minute stuff. Hopefully that gets fixed sometime this week [09:31:07] hashar: can't we run a git submodule init MobileApp by hand? [09:32:37] YuviPanda: clever idea, unfortunately the jenkins job is made to recheck all submodules :D [09:32:44] hah [09:32:47] making sure to erase any live hacks that might have been done on beta :D [09:32:55] good idea, generally :D [09:33:06] I made it so nothing could be done manually (at least in theory) [09:33:11] hmm, yeah [09:33:14] ensuring everything goes via Gerrit / review [09:33:16] since [09:33:23] it will eventually land on production [09:33:42] right [09:33:44] you don't want a live hack on beta to be forgotten when deploying in prod [09:33:49] yeah [09:33:52] and that force people to write feature switches [09:34:01] so the new experimental feature is deployed in prod but disabled [09:34:07] right [09:34:09] then switched on on beta [09:34:11] feature switches are nice [09:34:12] ci at best [09:34:14] yeah [09:34:26] we probably have been using them since day one [09:34:40] I gave a few talks about wikimedia CI / deployment [09:35:04] oh? where? [09:35:09] and basically the summary: how to don't care about deploying bad code because it can be reverted in a second [09:35:27] :D [09:35:31] nice! [09:35:33] s/reverted/disabled/ [09:35:44] talks to local groups or local startups [09:35:46] hashar: hopefully in a few years we can actually do continuous *deployment* :P [09:35:47] nothing fancy [09:35:51] rather than just integration [09:35:52] yeah that is the aim [09:36:02] we talked about it extensively with Greg back in the amsterdam hackaton last year [09:36:10] that is more or less a long term vision [09:36:20] yeah [09:36:27] we already deploy mw two or three times per week [09:36:28] hashar: how long term? [09:36:32] 3 years? 5? [09:36:33] and extensions more or less continuously [09:36:33] 1? [09:36:39] maybe next year [09:36:44] there is a lot to achieve first [09:36:54] daily deployment is probably closer [09:37:01] we need to sort out scap first [09:37:10] and fix up l10n messages taking 30-40 minutes to sync [09:37:10] git-deploy? [09:37:13] or whatever that is called now [09:37:15] + moar integration tests [09:37:48] ah git-deploy is now named either: Sartoris (old legacy name), Trebuchet (new fancy name which might not take up) [09:38:04] or currently either: git-deploy || Ryan deployment system [09:38:06] heh I bet we'll rename it at least once more [09:38:07] haha [09:38:07] I think 'Trebuchet deploy' came up [09:38:17] * YuviPanda waves at ori without the -l [09:38:28] * ori waves at YuviPanda without the -zz [09:38:36] :D [09:38:42] the cabal is waiting for ori to replace all of that with a zeromq python script [09:38:56] with bittorrent [09:39:01] i am a one trick pony, it's true [09:39:12] written in Ruby, because Ops love Ruby [09:39:19] but it's a good trick :P [09:39:41] iirc ops concern with ruby is not having to maintain yet another language facing internet users [09:40:36] hashar: also things like ruby people saying 'oh just use RVM' [09:41:13] ori: shouldn't you be sleeping? [09:42:20] i don't sleep, i busy-loop [09:42:48] sounds very power intensive [09:43:04] It isn [09:43:10] isn't 11 UTC yet, YuviPanda [09:43:18] heh [09:43:32] * Aaron|home likes having compiled https://github.com/MSOpenTech/redis and having ceph inside vagrant...all in windows [09:44:36] * Aaron|home needs to compile hiphop in one of the other 2 VMs for completeness [09:44:49] we don't have a hhvm vagrant role yet, do we? [09:44:58] (03PS1) 10Alexandros Kosiaris: Purge etherpad package [operations/puppet] - 10https://gerrit.wikimedia.org/r/106220 [09:45:00] we do, but it's a stub [09:45:12] hmm, I should probably help out there [09:45:24] would be great [09:45:39] configuring hhvm to run behind nginx takes a minute, but it's cheating [09:45:47] because we're going to be using apache in prod, at least initially [09:45:57] * YuviPanda googles [09:45:57] since we don't want to pile a web server migration on top [09:46:03] http://www.sebastien-han.fr/blog/2013/04/22/play-with-ceph-vagrant-box/ is pretty cool [09:46:13] hhvm has shiny fastcgi support [09:46:28] i couldn't get it to work, but i didn't try very hard, and i know some fixes were committed in the last few days [09:46:33] * Aaron|home has a list of changes need to get that working for swift/mw [09:46:36] *needed [09:46:57] Aaron|home: make it an MWV role [09:47:15] YuviPanda: erik b. has it running behind apache in https://gerrit.wikimedia.org/r/#/c/105834/ [09:47:28] ori: hmm, so it our end result would be nginx -> varnish -> apache -> (fastcgi) hhvm? [09:47:32] but that's using ProxyPass + HTTP, no fastcgi [09:47:54] YuviPanda: for SSL, yes. [09:48:09] hmm, right [09:48:30] https://bugzilla.wikimedia.org/show_bug.cgi?id=59793 [09:48:38] ughh...can somebody saw oauth? [09:48:45] it's not the sort of design you sketch out for yourself; it's the sort of design that makes sense for a distributed organization with different teams working on different parts of the stack [09:49:09] it still needs to be simplified IMO but i think that's uncontroversial; just a matter of when [09:49:11] * Aaron|home handles ori https://gerrit.wikimedia.org/r/#/c/105241/ [09:49:34] * ori looks [09:49:49] how come no one harasses aaron for being up? he's in sf too [09:50:04] * YuviPanda saws OAuth for Aaron|home [09:50:07] erm, "hands" not "handles" ;) [09:50:13] * Aaron|home needs to sleep [09:50:23] ori: heh, we should have a bot that does the nagging... [09:50:53] * Aaron|home is also leaning into high left arm in a bad position and it's going numb now [09:51:06] *his [09:51:25] go sleep Aaron|home [09:51:27] geez [09:51:31] * Aaron|home changes position [09:51:31] :-P [09:51:33] ah...better [09:52:08] ori: I wonder if the UID stuff could use shm if available [09:53:03] what would that buy you? [09:53:39] in theory though, the only difference would be it not getting caught up in the fsync daemon, which might make no difference [09:54:06] * Aaron|home was also thinking about getLocalReference() using that but was afraid of spamming it with too much stuff [09:54:20] * Aaron|home wonders if domas was joking with that suggestion or not [09:54:46] it would help for those copy scripts, since that's actually a fair amount of mbs [09:56:47] i have a vague sense that HHVM might change the rules of the game a little but dunno [09:56:59] now sleep! [09:57:07] You can't! [09:57:10] (03CR) 10Alexandros Kosiaris: [C: 032] Purge etherpad package [operations/puppet] - 10https://gerrit.wikimedia.org/r/106220 (owner: 10Alexandros Kosiaris) [09:57:11] It's not 11 UTC yet! [09:57:20] what happens at 11 UTC? [09:57:34] You get to bed at 11 UTC [09:57:57] ori: making fun of you; remember the day Nemo pinged you about why you're still working at 3 AM? :-) [09:58:04] ah, heh [09:58:24] yes, it's probably accurate [10:01:36] ori: https://gerrit.wikimedia.org/r/105677 [10:02:14] ori: i thought you were going to sleep :P [10:02:23] Nemo_bis: 01:45 configuring hhvm to run behind nginx takes a minute, but it's cheating [10:02:40] it takes more than a minute and it's not cheating, i was being hyperbolic [10:02:49] but it's still not exactly right [10:02:52] anyway, https://github.com/facebook/hhvm/issues/1437 worth following it seems [10:03:16] well there are measurable improvements [10:03:16] hmm, $wgServer bites again! [10:03:29] heh [10:04:16] * Aaron|home read "hyperbolic" as in "hyperbolic topology" [10:04:35] Nemo_bis: HHVM + nginx is a great setup; the only thing going against it is the fact that it'd be too hard to swap out Apache at the moment [10:04:52] ori: not for everyone though [10:05:16] hmm does wikiapiary have stats on how many use apache [10:05:21] Nemo_bis: i wasn't disparaging Nikerabbit's patch; it looks really cool [10:06:18] YuviPanda: also, have you seen ? [10:06:37] oh, someone set us up on Travis? [10:06:38] one failure, already fixed upstream: https://bugzilla.wikimedia.org/show_bug.cgi?id=55532 [10:06:49] quite nice! [10:06:59] ori: is that building with hhvm? [10:07:04] yeah [10:07:05] * ori nods [10:07:12] nice! [10:07:21] ori: did you test ebernhardson's patch? [10:07:27] https://github.com/wikimedia/mediawiki-core/blob/master/.travis.yml [10:07:30] YuviPanda: nope [10:08:09] ori: and I wasn't defending it, just wondering how the different usecases can help each other benefit from the experiments around hhvm [10:08:45] ori: hmm, okay. [10:08:46] i wonder if we should create a mailing list [10:08:55] there is one already.. [10:08:56] and an IRC channel [10:08:56] it'll probably die out eventually after the migration but it might still make sense [10:08:58] :P [10:09:27] rather, shouldn't there be a bugzilla keyword or at least whiteboard something for hhvm stuff [10:09:52] ori: http://lists.wikimedia.org/pipermail/hiphop/ [10:09:56] Nemo_bis: i was just discussing that with andre over at -dev [10:10:05] (or are you talking of something else= [10:10:07] oh [10:10:07] https://bugzilla.wikimedia.org/show_bug.cgi?id=40926#c5 [10:10:42] * Nemo_bis hadn't noticed the blocker [10:27:54] (03PS1) 10Alexandros Kosiaris: Remove hooper, eiximenis [operations/dns] - 10https://gerrit.wikimedia.org/r/106223 [10:30:26] !log screwed gallium by doing a chown -R jenkins-slave:jenkins-slave on /srv/ that includes the Zuul git repositories :( [10:30:32] Logged the message, Master [10:36:02] hashar: what is fatal: bad revision 'remotes/gerrit/origin' ? [10:38:56] (03CR) 10Dzahn: [C: 031] "thx for also killing eiximenis, was never sure where it originally started" [operations/dns] - 10https://gerrit.wikimedia.org/r/106223 (owner: 10Alexandros Kosiaris) [10:41:58] (03CR) 10Dzahn: "oh..hmm.. see https://twitter.com/byteofprash/status/15689761272373248 f.e., we seem to have some incoming links to this in Google.. worth" [operations/dns] - 10https://gerrit.wikimedia.org/r/106223 (owner: 10Alexandros Kosiaris) [10:42:29] akosiaris: grmbl.. we would break $some URL [10:43:20] correction, it was broken a long time, but links to it exist out there [10:44:16] mutante: which one ? [10:44:37] google: eiximenis.wikimedia.org -ganglia [10:45:20] https://twitter.com/byteofprash/status/15689761272373248 but not that many .. shrug [10:47:41] mutante: seems like all are etherpad pads [10:47:55] https://etherpad.wikimedia.org/p/MediaWikiNotes [10:48:04] akosiaris: yes [10:48:47] old etherpad was reachable under that so people used it, or it was just hostname before it got servicename as well [10:49:07] yeah [10:49:13] well two solutions I see (yoda talk) [10:49:15] before my time as well [10:49:27] 1) point eiximenis as a CNAME to etherpad [10:49:29] and now old etherpad is gone [10:49:44] 2) not care and see if people complain [10:50:00] re 2) they didnt complain all this time [10:50:05] exactly [10:50:08] it's not like it worked yesterday [10:50:18] re 1) good URLs dont change [10:50:22] hmmmm [10:50:41] I vote for 2 [10:51:07] it is called the shooting method akosiaris [10:51:15] leaning to 2, but it's a little weak because the "dont break URLs"mantra is strong [10:51:23] *shouting [10:51:30] but it's already done.. so ok ..2 [10:51:54] all better than remnants in our code that didnt work anyways :) [10:52:03] mutante: yeah but notice the "good" adjective in the mantra [10:52:11] hehe, fair [10:52:19] I 'd say a url having a host part inside ... not so good :-) [10:52:24] hostnames in URLs suck [10:52:29] yea [10:52:44] ok, kill, thx for cleanup :) [10:53:13] ok. I will merge this after decomming hooper and wiping it [10:53:24] thanks [10:53:27] cool [11:05:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 11:01:51 PM UTC [11:54:00] akosiaris: would manifests/misc/racktables.pp be appropriate to be a module"? [12:24:20] matanya: it is so small that I am tempted to say no. Anyway... there is an effort to migrate away from racktables so don't waste any time on it [12:24:20] to what akosiaris [12:27:21] ralph has been evaluated but has not received any positive reviews, servermon is a way forward (it is missing still functionality however) [13:01:44] !log jenkins: migrating jobs git url from integration.wikimedia.org to zuul.eqiad.wmnet {{bug|59774}} {{gerrit|106116}} [13:01:50] Logged the message, Master [13:04:43] !log "upgrading" pep8 package 1.4.6-1 --> 1.4.6-1.1 (simply provides python3-pep8) RT #6420 [13:04:50] Logged the message, Master [13:05:36] akosiaris: thank you for the pep8 update [13:09:41] :-) [13:12:56] (03PS1) 10MaxSem: Enable beta mobile diff on testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106236 [13:36:53] ori: cool [13:54:33] !log restarting Zuul, it lost track of some jobs and keep changes in its queues for no real reason :/ [13:54:39] Logged the message, Master [14:06:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 11:01:51 PM UTC [14:41:47] (03PS1) 10Springle: Expose max_user_connections var in mysql_multi instance .cnf [operations/puppet] - 10https://gerrit.wikimedia.org/r/106254 [14:44:07] (03PS2) 10Hashar: zuul: define push_change_refs has false [operations/puppet] - 10https://gerrit.wikimedia.org/r/105959 [14:48:43] DB errors [14:49:04] קפיצה אל: ניווט, חיפוש [14:49:04] אירעה שגיאה בשאילתה לבסיס הנתונים. שגיאה זו עלולה להעיד על באג בתוכנה. [14:49:04] פונקציה: IndexPager::buildQueryInfo (LogPager) [14:49:04] שגיאה: 1176 Key 'PRIMARY' doesn't exist in table 'logging' (10.64.16.30) [14:50:07] and in english: there was an error in the DB query: function: IndexPager::buildQueryInfo (LogPager) [14:50:19] error: 1176 Key 'PRIMARY' doesn't exist in table 'logging' (10.64.16.30) [14:50:48] probably my fault. s7 is testing a partitioned logging [14:51:43] springle: can't hide revisions [14:52:17] arg more FORCE INDEX [14:52:21] * springle sighs [14:56:43] (03PS1) 10Springle: depool db1041, forced index query errors during partitioning test [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106255 [14:58:15] (03CR) 10Springle: [C: 032] Expose max_user_connections var in mysql_multi instance .cnf [operations/puppet] - 10https://gerrit.wikimedia.org/r/106254 (owner: 10Springle) [15:02:12] (03PS1) 10Hashar: zuul: monitor gearman service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 [15:04:10] !log springle synchronized wmf-config/db-eqiad.php 'depool db1041, forced index query errors during partitioning test' [15:04:17] Logged the message, Master [15:07:03] (03CR) 10Dzahn: [C: 032] "root@gallium:~# /usr/lib/nagios/plugins/check_tcp -H 127.0.0.1 -p 4730 --timeout=2" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 (owner: 10Hashar) [15:09:00] tests are lagging again :( [15:11:34] for some reason Jenkins has tests being stuck: https://integration.wikimedia.org/ci/job/operations-puppet-doc/4052/console [15:12:54] how nice [15:13:29] yeah and the queue on left of https://integration.wikimedia.org/ci/ shows red jobs [15:13:33] i.e. they are stalled somehow [15:14:00] I really need to drop Jenkins [15:21:55] thanks springle [15:22:13] matanya: np. sorry about that [15:22:31] thought I'd found all the forced indexes to work around [15:23:40] i guess now you did :) [15:26:37] (03PS2) 10Hashar: zuul: monitor gearman service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 [15:30:16] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Better to use the nrpe::monitor_service definition." (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 (owner: 10Hashar) [15:30:37] ahhhhh [15:33:35] (03CR) 10Dzahn: [C: 031] zuul.pp retab + some puppet-lint fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/105958 (owner: 10Hashar) [15:34:40] (03CR) 10Dzahn: [C: 031] zuul: define push_change_refs as false [operations/puppet] - 10https://gerrit.wikimedia.org/r/105959 (owner: 10Hashar) [15:37:19] (03CR) 10Hashar: "bah I copy pasted the zuul check :( Will refactor in a new patchset" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 (owner: 10Hashar) [15:38:40] (03CR) 10Dzahn: "eh yeah, didn't think of the new method either, back then it was state of the art :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 (owner: 10Hashar) [15:39:07] mutante: akosiaris: I will replace the patch with akosiaris proposal [15:39:16] :) [15:39:49] (03CR) 10Springle: [C: 032] depool db1041, forced index query errors during partitioning test [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106255 (owner: 10Springle) [15:41:04] (03CR) 10Hashar: zuul: monitor gearman service (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 (owner: 10Hashar) [15:41:20] (03PS3) 10Hashar: zuul: monitor gearman service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 [15:42:44] cool :-). I will merge as soon as jenkins say ok :-) [15:43:18] we can also do the retabbing of zuul.pp if you want , hashar [15:43:23] lgtm [15:45:25] (03PS1) 10Hashar: zuul: switch to nrpe::monitor_service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106267 [15:45:29] ah shit [15:45:31] forgot the retab [15:45:34] gotta rebase :-] [15:48:58] (03PS4) 10Hashar: zuul: monitor gearman service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 [15:48:59] (03PS2) 10Hashar: zuul: switch to nrpe::monitor_service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106267 [15:49:19] (03CR) 10jenkins-bot: [V: 04-1] zuul: switch to nrpe::monitor_service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106267 (owner: 10Hashar) [15:49:58] (03PS5) 10Hashar: zuul: monitor gearman service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106256 [15:50:01] (03PS3) 10Hashar: zuul: define push_change_refs as false [operations/puppet] - 10https://gerrit.wikimedia.org/r/105959 [15:50:02] (03PS2) 10Hashar: zuul.pp retab + some puppet-lint fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/105958 [15:50:56] (03PS3) 10Hashar: zuul: switch to nrpe::monitor_service [operations/puppet] - 10https://gerrit.wikimedia.org/r/106267 [15:50:58] sometimes I hate gerrit [15:52:52] mutante: so https://gerrit.wikimedia.org/r/#/c/105958/ is the retab, easy enough [15:53:08] and I have rebased all the other changes on top of it [15:53:16] (03PS4) 10Dzahn: do not use generated .htaccess from Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 [15:53:20]