[00:03:14] (03CR) 10Qgil: [C: 031] Updated internal.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 (owner: 10RashiqAhmad) [00:04:29] (03CR) 10Qgil: [C: 04-1] "https://gerrit.wikimedia.org/r/#/c/102354/ is on its way to be merged." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100326 (owner: 10Gerrit Patch Uploader) [00:06:00] can anybody restart the parsoids? [00:06:11] service-restart did not work: https://gist.github.com/subbuss/22902cec7e8c90429c41/raw/68b011463579bf7fa5fef04de0187fb7ffc84fbb/gistfile1.txt [00:06:29] dsh -g parsoid service parsoid restart [00:07:11] or, using salt: salt -b 10% -G 'deployment_target:parsoid' parsoid.restart_parsoid parsoid [00:07:17] (from https://wikitech.wikimedia.org/wiki/Parsoid#Misc_stuff) [00:07:36] mutante ^^ [00:07:37] gwicke: do you have a preference which one? [00:07:40] i use [00:07:48] does it matter how fast it happens? [00:07:52] mutante: no, either should work [00:07:57] ok [00:08:01] ideally not all at once [00:08:11] both should do it serially [00:08:12] well, thats what i meant [00:08:16] ok [00:08:17] salt 10% at a time [00:08:22] dsh one at a time [00:08:54] if the error is with salt in general then dsh might be the only option [00:09:40] gwicke: doing salt [00:09:53] !log doing parsoid restart via salt, with the 10% option [00:10:10] Logged the message, Master [00:10:20] it says things about Executing runs [00:10:22] !log updated parsoid config to add support for tyv and min wikipedias [00:10:23] and return code 0 [00:10:26] lgtm , right [00:10:39] Logged the message, Master [00:10:43] seems to have worked [00:10:47] thanks! [00:10:47] great [00:10:49] yw [00:11:24] we'll have to fix that up next year [00:14:25] mutante, thanks [00:14:32] np [00:17:58] (03Abandoned) 10Aquifacae: Added new 48x48 resolution to wiktionary/en favicon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102554 (owner: 10Aquifacae) [00:19:07] (03PS1) 10Yurik: Bug 58676: Added logging for "Invalid message parameter" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102601 [00:20:39] mutante, are you syncing things? [00:21:59] (03PS1) 10Aquifacae: Made proper larger favicon wiktionary/en.ico favicon. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102602 [00:22:23] bd808: ping [00:22:36] paravoid: pong [00:22:41] yurik: no [00:22:42] I'm reading backlog [00:22:49] everything okay with varnish? [00:22:53] greg-g, could i push https://gerrit.wikimedia.org/r/#/c/102601/ out? it adds a special log for the message param bug [00:22:55] Ori and I are trying to figure out what I did wrong in my app [00:23:11] okay [00:23:16] POST requests aren't working from outside the nginx+varnish layer [00:23:32] They work from inside when I'm hitting apache directly [00:24:17] Varnish seems to sometimes or maybe always) convert the POST to GET when it polls the backend [00:24:51] The app returns a 404 for a GET request to the POST routes (which end in '.post') [00:25:07] Varnish caches that 404 page and replays it [00:25:21] So basically fail [00:26:05] My thinking is that I need to return headers that I'm returning and/or configure varnish differently for my app [00:29:24] The only varnish config we added to puppet for this is -- http://git.wikimedia.org/blob/operations%2Fpuppet/04f0e1fdd2e18aa2a81b451dafd6e56be5be91be/templates%2Fvarnish%2Fmisc.inc.vcl.erb#L16 [00:30:12] And I didn't do anything explicitly in the app or apache to communicate with varnish [00:30:31] All requests to the app set or carry cookies (php session) [00:30:44] That's what I know at this point :/ [00:30:58] (03PS1) 10Faidon Liambotis: Varnish misc: return pass for non-GETs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102605 [00:32:51] (03PS2) 10Faidon Liambotis: Varnish misc: return pass for non-GETs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102605 [00:33:11] (03CR) 10Faidon Liambotis: [C: 032] Varnish misc: return pass for non-GETs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102605 (owner: 10Faidon Liambotis) [00:33:18] (03CR) 10Faidon Liambotis: [V: 032] Varnish misc: return pass for non-GETs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102605 (owner: 10Faidon Liambotis) [00:35:43] paravoid: http://stackoverflow.com/a/13970262 ? [00:35:50] bd808: can you try again? [00:36:45] ori-l: his solution is also wrong [00:37:39] It's better. No 404. I'm not getting logged in, but let me check the app error logs [00:38:24] I'm in! [00:39:12] cool [00:39:28] Can you check something? https://scholarships.wikimedia.org/review/phase1 [00:39:42] ok? [00:39:42] That should redirect you to the /login page [00:39:52] it did [00:40:00] Ok. great. [00:40:02] you're redirecting from http to https, though [00:40:02] paravoid: <3 [00:40:03] thanks [00:40:06] (different bug) [00:40:10] i was flailing around [00:40:18] I just wanted to be sure that varnish was vary'ing on the cookie [00:40:35] if you vary on the cookie, it will, but please don't :) [00:40:53] for logged in users, send no-cache headers [00:41:16] Ok. I'll file a bug and get that added [00:43:21] and folks, I know this is a minor deployment [00:43:29] it was scheduled [00:43:34] I know [00:44:04] but I think you'd be served better if someone from ops was here to assist you with it [00:44:20] yes, i think that's a pretty safe assumption :) [00:44:21] and it's totally ops' failure that didn't say so before [00:44:42] but you should also aggressively say so too :P [00:45:17] well, i don't know if it's ops or platform, but it's not me :P [00:45:43] paravoid: Point taken. I knew you'd fix it in minutes but was kind of hoping we would at least be able to describe the problem [00:45:48] bd808: birth pangs aside, congrats on the successful deployment [00:45:54] indeed, congrats :) [00:46:20] bd808: it's not just that, I'm just sad that noone has really helped with you all the other stuff too [00:46:28] apaches, puppet etc. [00:47:37] I look forward to you and your team having more bandwidth. I've seen how busy things are just keeping the lights on [00:49:56] http://www.theregister.co.uk/2013/12/18/harvard_bomb_hoax_charge/ [00:50:26] ok, maybe there could have been a more appropriate channel for that link [00:50:56] but I just wanted to post it in case anyone still thinks that Tor protects your privacy [01:11:44] (03CR) 10Dzahn: [C: 032] install various Perl modules needed by Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/101174 (owner: 10Dzahn) [01:13:02] now that we just talked about zirconium i did that too [01:14:19] paravoid: i just talked to bd808 about just that and offered puppet/apache/ssl cert help and related [01:14:52] awesome :) [01:27:42] (03CR) 10Dzahn: "notice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully" [operations/puppet] - 10https://gerrit.wikimedia.org/r/101174 (owner: 10Dzahn) [01:29:22] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [01:29:22] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [01:38:26] (03CR) 10Dzahn: "guess what, once those are in use and we find no more issues, all of that functionality of BZ listed above while be served by actually mai" [operations/puppet] - 10https://gerrit.wikimedia.org/r/101174 (owner: 10Dzahn) [01:52:09] (03CR) 10Dzahn: "hey thanks for your comments, of course i can fix tab/space issue, just let me know PS1 or PS2 :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83768 (owner: 10Dzahn) [02:00:18] (03CR) 10Dzahn: "thanks Alex :) i love actual wiki style editing on old patch sets. then this one will definitely stay for the new year. i almost like how " [operations/puppet] - 10https://gerrit.wikimedia.org/r/96424 (owner: 10Dzahn) [02:04:41] (03PS1) 10Tim Landscheidt: Tools: Work around broken WMF package libvips-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 [02:09:37] anyone have any knowledge of these databases? https://bugzilla.wikimedia.org/show_bug.cgi?id=58680 [02:14:37] (03PS1) 10Tim Landscheidt: Tools: Fix quoting error [operations/puppet] - 10https://gerrit.wikimedia.org/r/102613 [02:16:19] springle: Reedy would be the best person to ask [02:16:45] !log LocalisationUpdate completed (1.23wmf7) at Thu Dec 19 02:16:45 UTC 2013 [02:17:07] Logged the message, Master [02:18:35] ok tnx paravoid [02:19:43] Haha [02:19:49] A few of them predate me [02:19:54] :) [02:19:58] I've answered about chunk of them [02:20:04] TimStarling: maybe ^ [02:20:17] Indeed, or brion would have some idea [02:20:43] Might be worth seeing when the newest revisions to them are/were [02:21:02] (03PS1) 10Dzahn: add second ssh key for Yuvi [operations/puppet] - 10https://gerrit.wikimedia.org/r/102614 [02:21:19] where does this list come from? [02:21:44] me listing databases and grepping *dblist [02:22:04] show databases on s3 [02:22:16] | wikiconfig | [02:22:16] | wikidatawiki_old | [02:22:16] | wikimania | [02:22:16] | wikimania2005wiki | [02:22:17] deleted.dblist is just for documentation, it's not used by anything [02:22:33] I added databases to it when I deleted things from all.dblist, but others probably didn't do the same [02:22:33] Wikimania and wikimaniawiki [02:23:21] so that is the most likely explanation for comcomwiki, langcomwiki and noboardwiki [02:23:40] wikimania looks to be an old registration system [02:23:52] which are wikis for the communications committee, language committee, and a combined wiki for chapters without a board [02:23:57] wikimaniawiki is empty [02:24:15] dkwiki, zh_cnwiki, zh_twwiki probably have a common cause [02:24:40] since they are all aliases for languages that exist as some other code [02:25:06] (03CR) 10coren: [C: 032] "This makes 1.3 senses." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [02:25:13] wikidatawiki_old is the pre-s6 versio5 [02:25:19] *s5 version [02:25:30] (03PS2) 10coren: Tools: Add key for MariaDB repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/102585 (owner: 10Tim Landscheidt) [02:25:53] (03CR) 10coren: [C: 032] "Key is verified." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102585 (owner: 10Tim Landscheidt) [02:26:02] l10nwiki seems to predate language cdb files... [02:26:46] (03PS2) 10coren: Tools: Work around broken WMF package libvips-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [02:26:50] dkwiki actually has 1735 rows in its page table [02:27:18] (03CR) 10coren: [C: 032] "Still makes sense after rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [02:27:19] wikiconfig is december 2008 testing by Brion it seems [02:27:26] so I guess that was a real wiki at some point [02:27:28] (03CR) 10Faidon Liambotis: [C: 04-2] "If it's broken, fix it, don't workaround it." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [02:27:32] | 3 | testwiki | 20081224020545 | 0 | Kylu | testwiki | Enabling special:nuke | [02:27:32] | 4 | testwiki | 20081224042537 | 1 | Kylu | testwiki | undoing changes per req | [02:27:42] oh, I bet it was created by the script [02:28:07] devwikiinternal has no revision table :D [02:28:13] and the people actually started putting articles in it, and then it was merged with dawiki [02:28:17] and then deleted [02:28:32] paravoid: o_O I would much rather pin to the distro version than randomly guess how to build an unprovided dependency and how it might have been patched. [02:28:43] (03PS3) 10Tim Landscheidt: Tools: Work around broken WMF package libvips-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 [02:28:50] zh_cnwiki is pre-1.5 [02:28:53] (03CR) 10jenkins-bot: [V: 04-1] Tools: Work around broken WMF package libvips-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [02:29:09] Coren or others, would be appreciated by Yuvi and I if you review https://gerrit.wikimedia.org/r/#/c/102614/1 [02:29:11] Coren: why randomly guess? [02:29:13] 1 row in its page table [02:29:21] I mean cur table [02:29:48] i'm about to leave and made it my rule to not merge access stuff without another +1, he isn't immediately blocked, but later he kind of will be [02:30:18] (03PS2) 10coren: add second ssh key for Yuvi [operations/puppet] - 10https://gerrit.wikimedia.org/r/102614 (owner: 10Dzahn) [02:30:20] Coren: investigate if/how the package in apt is broken & provide a fix? :) [02:30:38] Coren: thanks! [02:31:31] !log LocalisationUpdate completed (1.23wmf6) at Thu Dec 19 02:31:31 UTC 2013 [02:31:46] Logged the message, Master [02:32:04] paravoid: I know /how/ it's broken. It's patched locally and depends on a local package for which a .deb isn't provided. I have no idea whether that was patched, and introducing the wrong build in the apt repo can break prod. [02:33:19] (03CR) 10coren: [C: 032] "It's either Yuvi, or an identical copy. (Although having /both/ would be even better then!)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102614 (owner: 10Dzahn) [02:34:38] Coren: try imagining that someone wants to install "libvips-dev" in production and you'd like to provide a fix for that :) [02:35:46] paravoid: So unless someone produces the .deb that is actually in prod, or the source to rebuild it, I'm definitely not going to add a random version of libtiff5 to the reprepo! [02:36:04] (03CR) 10Tim Landscheidt: "@Faidon Liambotis: AFAICS the WMF package belongs to a family of packages used by manifests/imagescaler.pp. Fixing this is *way* outside " [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [02:36:20] I'm suggesting that this someone should be you [02:36:25] paravoid: Well, my first order of business would be to try to find who deployed a package that depends on a manual dpkg -i in prod to beat some sense into them. :-) [02:37:43] the vips we run in production is likely a backport; binary dependencies get automatically generated, so it's likely that in a precise build environment it depends on libtiff4, rather than libtiff5 [02:38:01] build-dependencies can't be automatically generated, though, so the upstream package had libtiff5-alt-dev hardcoded [02:38:22] and whoever did the backport didn't notice, and because we don't use the -dev in production it was never noticed until now either [02:39:07] so the bug is probably a bad backport and the fix is a sed -i s/libtiff5-alt-dev/libtiff4/4 debian/control [02:39:20] and a rebuild and reprepro include [02:39:38] paravoid: That seems likely enough. How confident are you that it's /just/ a backport? [02:39:47] look at debian/changelog? [02:39:52] fetch the source from apt? [02:39:54] etc. [02:40:15] or, well [02:40:19] https://gerrit.wikimedia.org/r/#/q/project:operations/debs/vips,n,z [02:40:22] even [02:40:58] Ah, yes. If we /do/ have the debs source that does help. :-) [02:41:17] we always have sources, usually in apt, most times in git too [02:41:35] and if we don't, it's bad enough to investigate what was changed and add them/rebuild [02:41:51] (03CR) 10Krinkle: "Pressed Publish instead of Publish and submit? (this repository isn't merged by Jenkins)" [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/63851 (owner: 10Mark Bergsma) [02:41:51] https://gerrit.wikimedia.org/r/#/c/63853/1/debian/control is the commit that changed the B-Dep, but not the -dev package's dep [02:42:04] line 4 was changed, but not 35 [02:42:07] (03PS4) 10Tim Landscheidt: Tools: Work around broken WMF package libvips-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 [02:42:29] (this would be the investigation I was referring to...) [02:42:59] (03CR) 10Andrew Bogott: [C: 032] "Good catch!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102613 (owner: 10Tim Landscheidt) [02:43:02] (03PS2) 10Andrew Bogott: Tools: Fix quoting error [operations/puppet] - 10https://gerrit.wikimedia.org/r/102613 (owner: 10Tim Landscheidt) [02:44:45] (03CR) 10Andrew Bogott: [C: 032] Tools: Fix quoting error [operations/puppet] - 10https://gerrit.wikimedia.org/r/102613 (owner: 10Tim Landscheidt) [02:47:49] Wait what? Mark just blindly changed the dependency from libtiff5 to libtiff4 and rebuilt? The -dev package can't be done this way, it relies on -alt-dev which has its include files in different locations. [02:49:01] I mean, you can change the dependency, but then you wouldn't be able to build anything with the resulting package. [02:49:19] how come? [02:49:23] springle: replied on the bug [02:50:16] paravoid: the 5-alt-dev package installs the header files in an out of include path location to avoid breaking the 4-dev headers; but then it hardcodes a lot of -I to work around it. [02:50:37] (Yes, that libtiff transition is a pain because they are mutually incompatible and can't live next to each other) [02:50:40] TimStarling: Reedy: thanks! [02:50:55] what hardcodes those -Is? [02:51:11] it probably all happens via pkgconfig [02:51:31] so vips will probably just work with libtiff4 [02:51:39] I mean, libvips itself built [02:52:10] (and libvips-tools as well) [02:52:13] Yeah, but then it uses the includes in its src tree; not those it expects on the system. [02:52:58] Hm. libvips-tools is something else. I'm surprised it built at all; and is probably a sign that it /does/ use pkgconfig. [02:53:09] Welp, it's worth a try. [02:53:18] it's built from the same source [02:53:34] Ah, yes, I see it in the control file. [02:53:43] So that may or may not be a good sign. :-) [02:53:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 19 02:53:46 UTC 2013 [02:54:02] Logged the message, Master [02:54:17] (Actually, I'm surprised it builds at all; there aren't that many API changes between 3.X and 4.0, but enough that most things break) [02:54:45] And yes, libtiff4 installs 3.9; libtiff5 installs 4.0. Yeay version madness. [02:54:53] srsly? [02:54:58] * Coren nodsnods. [02:55:42] Hang on, I remember an email on the topic from this spring on debian-release. Lemme try to find it. [02:56:18] paravoid: Read http://lists.debian.org/debian-release/2013/05/msg00127.html and look into the eyes of madness. [02:56:22] (03PS1) 10Springle: previously missed deleted wikis, bug 58680 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102615 [02:56:53] (03CR) 10Springle: [C: 032] previously missed deleted wikis, bug 58680 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102615 (owner: 10Springle) [02:57:50] that's way, way ahead of precise [02:58:05] so, anyway, libvips built fine with libtiff4 [02:58:35] !log springle synchronized database lists files: [02:58:37] Yeah, it's a reasonable assumption that -dev will survive; like I said "worth a try" [02:58:46] yup [02:58:49] Logged the message, Master [02:59:11] single line fix in gerrit + d/changelog bump, build, reprepro include [02:59:47] I wish I knew why it was backported in the first place though;. [03:00:35] it's easy to find out, apt-cache policy libvips-dev is an easy give-away [03:00:50] and /usr/share/doc/libvips-dev/changelog.Debian.gz has details [03:06:29] (03PS1) 10coren: Fix dependency libtiff5-alt-dev -> libtiff4-dev [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 [03:07:15] (03PS2) 10Faidon Liambotis: Fix dependency libtiff5-alt-dev -> libtiff4-dev [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 (owner: 10coren) [03:07:23] added the Bug [03:07:38] Ah; good point. [03:07:46] (03CR) 10Faidon Liambotis: [C: 032] Fix dependency libtiff5-alt-dev -> libtiff4-dev [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 (owner: 10coren) [03:07:58] do you know how to build packages? [03:08:32] In general. I was about to go hunt for documentation on our own process. [03:08:46] use a clean chroot using pbuilder [03:09:22] https://wikitech.wikimedia.org/wiki/Pbuilder [03:09:23] heh, nice [03:10:04] I guess we have plenty of docs on wikitech [03:10:08] let me know if I can help :) [03:10:18] "Start by getting your hands on a Labs instance with a misc::package-builder Puppet role". Do we have one dedicated to this task? [03:10:27] I don't know [03:10:36] * Coren goes to look for one. [03:10:48] I'm not going to build a new one just because. [03:11:01] I do a lot of building packages so I have my own machine outside of labs that has a bazillion chroots [03:18:26] (03PS3) 10coren: Fix dependency libtiff5-alt-dev -> libtiff4-dev [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 [03:18:43] Wrong number of zeroes [03:20:55] !log starting schema changes for gerrit 101170 page_links_updated [03:21:10] Logged the message, Master [03:23:17] Meh, I'll just build on my box [03:28:45] (03PS2) 10Aquifacae: Made proper larger favicon wiktionary/en.ico favicon. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102602 [03:45:08] (03CR) 10TTO: "Are you sure you don't want to specify the old name as a namespace alias for the project namespace? There are probably lots of incoming li" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102482 (owner: 10Ebrahim) [03:49:27] !log Reloading Zuul to deploy I5b759c388db5e5a [03:49:41] Logged the message, Master [04:30:22] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [04:30:22] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [05:06:42] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:10:42] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [05:26:29] (03CR) 10Matanya: "None, I did it by hand. This is the best way to find errors." [operations/puppet] - 10https://gerrit.wikimedia.org/r/98377 (owner: 10Matanya) [06:02:14] (03CR) 10Tim Landscheidt: "@Matanya: Maybe, but I can't execute you in a pre-commit hook :-). Thanks." [operations/puppet] - 10https://gerrit.wikimedia.org/r/98377 (owner: 10Matanya) [06:05:21] (03CR) 10Tim Landscheidt: "Ah, okay, just found puppet-lint (a "lint" for "puppet" - that's SEO). I'll play around with that." [operations/puppet] - 10https://gerrit.wikimedia.org/r/98377 (owner: 10Matanya) [06:05:23] (03CR) 10Matanya: "you can run puppet-lint in a pre-commit, and prevent from committing if errors or warnings found, unless overridden." [operations/puppet] - 10https://gerrit.wikimedia.org/r/98377 (owner: 10Matanya) [06:27:44] SpecialMobileHistory::doQuery 10.64.0.6 0 SELECT * FROM `revision` FORCE INDEX (page_timestamp) WHERE rev_user = '0' ORDER BY rev_timestamp DESC LIMIT 51 [06:27:48] springle: ?? [06:28:39] Aaron|home: ? [06:28:50] that looks pretty silly [06:28:55] * Aaron|home wonders who to blame [06:29:09] oh. yes it does [06:51:04] (03PS1) 10Matanya: ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 [06:52:11] (03CR) 10jenkins-bot: [V: 04-1] ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [06:54:05] (03PS2) 10Matanya: ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 [06:54:48] (03CR) 10jenkins-bot: [V: 04-1] ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [06:59:14] (03PS1) 10Tim Landscheidt: Tools: Resolve virtual package requirements [operations/puppet] - 10https://gerrit.wikimedia.org/r/102630 [07:16:13] (03PS3) 10Matanya: ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 [07:16:56] (03CR) 10jenkins-bot: [V: 04-1] ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [07:21:41] (03PS4) 10Matanya: ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 [07:22:25] (03CR) 10jenkins-bot: [V: 04-1] ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [07:27:37] (03PS5) 10Matanya: ldap : lint cleanup [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 [07:31:22] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [07:31:22] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [07:41:52] (03CR) 10Ebrahim: "It is just trivial for robots to fix, be sure we won't leave them anyway." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102482 (owner: 10Ebrahim) [07:46:23] (03PS1) 10Faidon Liambotis: Varnish: switch to W3C standard headers for ESI [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 [07:48:43] (03CR) 10Faidon Liambotis: "Yuri, let's do this before we deploy ESI in production and are in need of a three step transition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 (owner: 10Faidon Liambotis) [07:53:43] (03Abandoned) 10Faidon Liambotis: Tools: Work around broken WMF package libvips-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/102609 (owner: 10Tim Landscheidt) [08:10:51] (03CR) 10Ebrahim: "In the other word, نسک is not really understandable nowadays so keeping ویکی‌نسک links just make confusion." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102482 (owner: 10Ebrahim) [08:53:52] PROBLEM - MySQL Slave Delay on db69 is CRITICAL: CRIT replication delay 313 seconds [08:54:32] PROBLEM - MySQL Replication Heartbeat on db69 is CRITICAL: CRIT replication delay 339 seconds [08:56:32] RECOVERY - MySQL Replication Heartbeat on db69 is OK: OK replication delay 42 seconds [08:56:52] RECOVERY - MySQL Slave Delay on db69 is OK: OK replication delay 0 seconds [08:57:42] (03Abandoned) 10Matanya: imagescaler: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/95440 (owner: 10Matanya) [08:58:33] (03Abandoned) 10Matanya: mail.pp: change exec to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91573 (owner: 10Matanya) [09:05:25] (03PS2) 10Yurik: Varnish: switch to W3C standard headers for ESI [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 (owner: 10Faidon Liambotis) [09:06:06] yurik: any news regarding: https://gerrit.wikimedia.org/r/#/c/92288/ ? [09:06:11] yurik: that was the period of the sentence. [09:06:37] paravoid, yes, but it was making gerrit incorrectly highlight it as part of the URL :) [09:07:39] its just like the famous problem (when writing text in parenthesis, should we close it after a smily? :)) [09:09:16] matanya, i'm fine with removing that code - i think it was paravoid who put it there [09:10:52] * yurik is reading edge architecture spec... [09:11:49] yurik: I was, it was doubt over of our TTL the reason we haven't merged matanya's patch, though [09:12:00] and you had promised you were going to check :) [09:12:17] bleh, could i have a secretary please? [09:21:20] paravoid, i'm really not an expert on how the default caching policy in wiki works (it doesn't seem like we set it anywhere in zero except for ESI) [09:22:02] looking at some other settings... [09:24:13] paravoid: would you recomment moving nfs to a module? [09:26:09] matanya, paravoid, it seems everywhere i looked that we use 31 days for maxage, i say go for it [09:43:10] (03CR) 10Ebrahim: "I don't call it "discussed", Persian Wikipedia users even wanted changing the not familiar for Persian typeface to a looking better one bu" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102454 (owner: 10Ebrahim) [09:50:32] RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [09:50:33] !log powercycled cp1065, unresponsive, no messages on mgmt console [09:51:03] Logged the message, Master [09:53:02] PROBLEM - Varnish HTTP text-backend on cp1065 is CRITICAL: Connection refused [09:54:50] apergos: seems you killed cp1065 :( [09:55:01] wait [09:55:01] no [09:55:07] it has been down for hours [09:55:08] I should go back to bed… sorry [09:55:42] where by 'down' I mean no response even on mgmt, locked up [09:56:51] maybe you need your morning hot chocolate [09:58:02] RECOVERY - Varnish HTTP text-backend on cp1065 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.004 second response time [10:07:19] (03CR) 10Yurik: Varnish: switch to W3C standard headers for ESI (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102633 (owner: 10Faidon Liambotis) [10:08:57] paravoid, ^ [10:12:04] (03CR) 10Yurik: [C: 031] "I searched the WMF settings - seems like it is set to ~30 days at the most." [operations/puppet] - 10https://gerrit.wikimedia.org/r/92288 (owner: 10Matanya) [10:25:06] apergos: yup basically haven't noticed you were powercycling cp1065 hehe [10:25:15] had a coffee and feeling awake now [10:25:55] more beta parsoid? or are you onto something else today? [10:32:22] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [10:32:22] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [10:32:58] apergos: ah beta parsoid, havent checked the job [10:33:21] + I found out they migrated the Parsoid code repository from mediawiki/extensions/Parsoid to /Parsoid [10:33:22] :D [10:33:25] (03CR) 10Umherirrender: "Time for merge of this patch set should now be reached." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 (owner: 10Dereckson) [10:37:44] not an extension? [10:58:23] (03PS1) 10Mark Bergsma: Introduce LVS role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/102638 [11:01:02] (03PS2) 10Mark Bergsma: Introduce LVS role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/102638 [11:19:11] (03PS3) 10Mark Bergsma: Introduce LVS role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/102638 [11:22:46] apergos: yup parsoid migrated to /mediawiki/services/parsoid.git :/ [11:23:51] ah [11:23:57] what other services do we have I wonder [11:32:55] mathoid :-D [11:35:03] there is some discussion about migrating the OCaml to some backend API service [11:35:07] (03PS2) 10Siebrand: Bug 58676: Added logging for "Invalid message parameter" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102601 (owner: 10Yurik) [11:35:45] (03CR) 10Mark Bergsma: [C: 032] Introduce LVS role class [operations/puppet] - 10https://gerrit.wikimedia.org/r/102638 (owner: 10Mark Bergsma) [11:38:09] (03PS1) 10Mark Bergsma: Use root level lvs::balancer [operations/puppet] - 10https://gerrit.wikimedia.org/r/102641 [11:39:25] (03CR) 10Mark Bergsma: [C: 032] Use root level lvs::balancer [operations/puppet] - 10https://gerrit.wikimedia.org/r/102641 (owner: 10Mark Bergsma) [11:47:14] camloid [12:03:18] (03PS1) 10Mark Bergsma: Attempt to fix undefined param in dns::recursor [operations/puppet] - 10https://gerrit.wikimedia.org/r/102644 [12:04:23] (03CR) 10Mark Bergsma: [C: 032] Attempt to fix undefined param in dns::recursor [operations/puppet] - 10https://gerrit.wikimedia.org/r/102644 (owner: 10Mark Bergsma) [12:06:14] (03PS1) 10Mark Bergsma: Fix parameter name [operations/puppet] - 10https://gerrit.wikimedia.org/r/102651 [12:06:34] (03CR) 10Mark Bergsma: [C: 032 V: 032] Fix parameter name [operations/puppet] - 10https://gerrit.wikimedia.org/r/102651 (owner: 10Mark Bergsma) [12:10:04] (03PS1) 10Mark Bergsma: Fix name conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/102652 [12:11:05] (03CR) 10Mark Bergsma: [C: 032 V: 032] Fix name conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/102652 (owner: 10Mark Bergsma) [12:22:35] (03PS1) 10Mark Bergsma: Attempt to fix new class resource conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/102653 [12:24:38] (03CR) 10Mark Bergsma: [C: 032] Attempt to fix new class resource conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/102653 (owner: 10Mark Bergsma) [12:29:53] (03PS1) 10Mark Bergsma: Remove duplicate definition of add_ip6_mapped [operations/puppet] - 10https://gerrit.wikimedia.org/r/102654 [12:31:01] (03CR) 10Mark Bergsma: [C: 032] Remove duplicate definition of add_ip6_mapped [operations/puppet] - 10https://gerrit.wikimedia.org/r/102654 (owner: 10Mark Bergsma) [12:38:10] (03PS1) 10Mark Bergsma: LVS: remove lvs::static_labs_ips [operations/puppet] - 10https://gerrit.wikimedia.org/r/102656 [12:39:13] (03CR) 10Mark Bergsma: [C: 032] LVS: remove lvs::static_labs_ips [operations/puppet] - 10https://gerrit.wikimedia.org/r/102656 (owner: 10Mark Bergsma) [12:50:01] (03PS1) 10Mark Bergsma: LVS: rename monitor_service definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/102665 [12:51:14] (03CR) 10Mark Bergsma: [C: 032] LVS: rename monitor_service definitions [operations/puppet] - 10https://gerrit.wikimedia.org/r/102665 (owner: 10Mark Bergsma) [12:57:13] (03PS1) 10Mark Bergsma: LVS: remove lvs::balancer dependencies on lvs::configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/102666 [13:01:59] (03PS1) 10Mark Bergsma: Fix lvs::monitor_service_custom name [operations/puppet] - 10https://gerrit.wikimedia.org/r/102667 [13:02:00] (03CR) 10Mark Bergsma: [C: 032] LVS: remove lvs::balancer dependencies on lvs::configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/102666 (owner: 10Mark Bergsma) [13:03:07] (03CR) 10Mark Bergsma: [C: 032] Fix lvs::monitor_service_custom name [operations/puppet] - 10https://gerrit.wikimedia.org/r/102667 (owner: 10Mark Bergsma) [13:05:16] (03PS1) 10Mark Bergsma: LVS: Add missing parameter $lvs_class_hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/102668 [13:06:21] (03CR) 10Mark Bergsma: [C: 032] LVS: Add missing parameter $lvs_class_hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/102668 (owner: 10Mark Bergsma) [13:08:12] (03PS1) 10Mark Bergsma: LVS: Really add parameter $lvs_class_hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/102669 [13:08:30] (03CR) 10Mark Bergsma: [C: 032 V: 032] LVS: Really add parameter $lvs_class_hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/102669 (owner: 10Mark Bergsma) [13:09:36] holy shit [13:09:44] do I really need 5 commits to get a few parameters passed? ;) [13:09:51] :-D [13:10:55] (03PS1) 10Mark Bergsma: LVS: Pass parameter $pybal_global_options as well [operations/puppet] - 10https://gerrit.wikimedia.org/r/102670 [13:11:32] * sdehaan remembers to file a ticket for the search api. [13:11:46] seeing some fairly high latency for the SMS & USSD bits of Wikipedia Zero [13:12:05] (03CR) 10Mark Bergsma: [C: 032] LVS: Pass parameter $pybal_global_options as well [operations/puppet] - 10https://gerrit.wikimedia.org/r/102670 (owner: 10Mark Bergsma) [13:16:52] (03PS1) 10Mark Bergsma: LVS: tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/102671 [13:18:24] (03CR) 10Mark Bergsma: [C: 032] LVS: tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/102671 (owner: 10Mark Bergsma) [13:26:48] got a cache question! I would like to use brewster as a web cache when fetching ruby gems [13:27:01] but the fetch is done using https so varnish does not cache the content [13:27:14] I guess for security purpose. Was wondering if we could ask varnish to cache nonetheless [13:27:22] to do illegal stuff? ;) [13:28:25] the use case is labs instance fetching ruby gems over https [13:28:45] I would like to save up the time to fetch from https://rubygems.* thus my idea was to use brewster as a "local" cache [13:33:22] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [13:33:22] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [13:34:53] maybe I should go with http :D [13:35:45] (03PS1) 10Mark Bergsma: LVS: Convert to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102673 [13:36:03] i don't think labs should depend on brewster for such a thing [13:36:19] (03CR) 10jenkins-bot: [V: 04-1] LVS: Convert to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102673 (owner: 10Mark Bergsma) [13:36:57] hashar: i'm still having an issue with the puppet job runner … [13:37:04] it seems like its holding jobs in the queue until another gwtoolset batch job is kicked off. [13:37:06] (03PS2) 10Mark Bergsma: LVS: Convert to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102673 [13:37:07] then it runs the previous jobs and keeps the current one in queue. [13:37:23] any idea on why that might be happening or how to change it? [13:38:03] or anyone else online that may know something about the puppet configured job runner? [13:38:17] at the moment i'm referring to the one on beta commons [13:42:26] (03CR) 10Mark Bergsma: [C: 032] LVS: Convert to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102673 (owner: 10Mark Bergsma) [13:49:42] dan-nl: not sure [13:49:54] I can just confirm a job is hold in the queue :( [13:49:55] $ mwscript showJobs.php --wiki=commonswiki --list [13:49:55] gwtoolsetUploadMetadataJob User:Dan-nl/GWToolset/Metadata_Batch_Job/52b2f5de546c8 attempts=1 user-name=Dan-nl whitelisted-post=array(39) jobReleaseTimestamp=1387460122 status=unclaimed [13:50:15] gwtoolsetUploadMetadataJob: 1 queued; 0 claimed (0 active, 0 abandoned) [13:50:17] aka it is in the queue [13:50:32] PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100% [13:52:11] dan-nl: yeah the job is not found for some reason :( [13:52:18] mwscript nextJobDB.php --wiki=commonswiki --type gwtoolsetUploadMetadataJob [13:52:18] gives me nothing [13:52:44] should yields 'commonswiki' [13:56:46] dan-nl: and looking at the nextJobDB.php code, it invokes JobQueueAggregator::singleton()->getAllReadyWikiQueues() [13:56:58] which gives a list of job types and each databases having such types of jobs pending [13:57:02] there is now gwtoolset in there [13:59:31] dan-nl: that sounds like a bug in nextJobDB [14:00:08] it invokes JobQueueAggregator::singleton()->getAllReadyWikiQueues(); which gives something different than JobQueueGroup::singleton(); used in showJobs.php [14:00:15] so that sounds like a bug in JobQueueAggregator [14:00:30] and I must confess I have never read that code nor I know what this class is about :( [14:00:59] oh boy [14:02:46] is this part of an extension? ParsoidCacheUpdateJobOnDependencyChange [14:08:23] that one should be Parsoid [14:12:18] (03PS1) 10Dan-nl: gwtoolset-runners [operations/puppet] - 10https://gerrit.wikimedia.org/r/102675 [14:12:59] hashar, i have an idea. atm all of the jobs are listed on one line. with https://gerrit.wikimedia.org/r/102675 i have listed them separately [14:13:29] maybe that will worl [14:13:32] *work [14:13:35] can you merge that? [14:13:55] one day I will have to rewrite that script to some python [14:14:49] dan-nl: it is not going to solve the issue imho [14:15:20] the first parameter of the shell function runJobsLoopService ends up being passed to: php MWScript.php nextJobDB.php --types="" [14:15:39] if I run that passing only one type, the script does not return any DB [14:15:58] which is because nextJobDB.php calls JobQueueAggregator::singleton()->getAllReadyWikiQueues(); which does not have the gwtoolset jobs for some reason [14:16:45] hmm … is this something for AaronSchulz to look into then? [14:17:21] also, odd because if i kick off a 2nd batch upload it does seem to pick up the job waiting in the queue [14:17:25] http://paste.debian.net/71775/ [14:17:48] that list the job types and the db having some pending [14:18:29] and here is the result of showJobs : http://paste.debian.net/71776/ [14:18:40] which does show a gwtoolsetUploadMetadataJob is in the queue [14:18:58] strange [14:19:08] I have Zero idea what JobQueueAggregator is doing, it might be misconfigured and not finding the gwtoolset jobs [14:21:36] might want to fill in a bug saying nextJobDB.php does not find some specific type although showJobs.php does show them as being in the queue [14:21:42] ccing aaron [14:22:58] or i may need to change the gwtoolset config … looking into how parsoid is configured [14:24:26] (03CR) 10Hashar: [C: 04-1] "the first parameter of the shell function runJobsLoopService ends up being passed to: php MWScript.php nextJobDB.php --types=""" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102675 (owner: 10Dan-nl) [14:24:32] dan-nl: replied on the gerrit change [14:24:47] thanks [14:24:54] can't really look further sorry ;:-( [14:25:19] thanks for your help with at least this [14:25:25] i'll abandon that other change [14:33:27] dan-nl: yeah I am almost sure it is an issue somewhere in the JobQueueAggregator [14:33:36] not sure how it looks up for db though [14:33:43] might want to copy paste my Gerrit comment in bugzilla [14:34:09] http://paste.debian.net/71775/ does show that commonswiki is known somehow [14:34:28] there is a htmlCacheUpdate job there [14:34:29] but somehow your jobs are not shown :/ [14:35:06] thanks for your research i really appreciate it! [14:35:19] i'll try and contact AaronSchulz tonight and see if he can sort it out further [14:40:22] PROBLEM - Puppet freshness on lvs3 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 11:39:39 AM UTC [14:42:22] PROBLEM - Puppet freshness on lvs4 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 11:42:07 AM UTC [14:43:40] (03PS1) 10Mark Bergsma: role/cache: tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/102681 [14:44:59] (03CR) 10Mark Bergsma: [C: 032] role/cache: tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/102681 (owner: 10Mark Bergsma) [14:47:00] mutante: is is https://bugzilla.wikimedia.org/ the place I'd file a ticket for high latency on the search apis? [14:54:22] (03PS1) 10Mark Bergsma: Add empty LVS service IP hashes for pmtpa [operations/puppet] - 10https://gerrit.wikimedia.org/r/102683 [14:55:57] (03CR) 10Mark Bergsma: [C: 032] Add empty LVS service IP hashes for pmtpa [operations/puppet] - 10https://gerrit.wikimedia.org/r/102683 (owner: 10Mark Bergsma) [14:57:52] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Thu Dec 19 14:57:45 UTC 2013 [14:58:22] !jenkins parsoidsvc-parse-tool-check [14:58:22] https://integration.wikimedia.org/ci/job/parsoidsvc-parse-tool-check [15:01:35] (03CR) 10Tim Landscheidt: "The change looks good to me, but given the connection to the image scalers, I'll leave reviewing it properly for others :-)." [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 (owner: 10coren) [15:11:59] (03PS1) 10Mark Bergsma: Fix lvs::monitor_service_custom name [operations/puppet] - 10https://gerrit.wikimedia.org/r/102691 [15:12:02] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Thu Dec 19 15:11:53 UTC 2013 [15:13:07] (03CR) 10Mark Bergsma: [C: 032] Fix lvs::monitor_service_custom name [operations/puppet] - 10https://gerrit.wikimedia.org/r/102691 (owner: 10Mark Bergsma) [15:19:47] (03PS1) 10Mark Bergsma: Fix ulsfo LVS monitoring duplicate definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/102694 [15:21:01] (03CR) 10Mark Bergsma: [C: 032] Fix ulsfo LVS monitoring duplicate definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/102694 (owner: 10Mark Bergsma) [15:33:01] (03PS1) 10Mark Bergsma: Memcached: reformat [operations/puppet] - 10https://gerrit.wikimedia.org/r/102696 [15:33:01] (03PS1) 10Mark Bergsma: Memcached: split off ganglia monitoring into a sub class [operations/puppet] - 10https://gerrit.wikimedia.org/r/102697 [15:33:02] (03PS1) 10Mark Bergsma: Memcached: tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/102698 [15:34:35] (03CR) 10Mark Bergsma: [C: 032] Memcached: reformat [operations/puppet] - 10https://gerrit.wikimedia.org/r/102696 (owner: 10Mark Bergsma) [15:35:24] (03CR) 10Qgil: [C: 04-1] "The version proposed is visibly more fuzzy than the current one. Look both images at 400% and you will see the problem clearly. Since nobo" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102602 (owner: 10Aquifacae) [15:35:33] (03CR) 10Mark Bergsma: [C: 032] Memcached: split off ganglia monitoring into a sub class [operations/puppet] - 10https://gerrit.wikimedia.org/r/102697 (owner: 10Mark Bergsma) [15:36:09] (03CR) 10Mark Bergsma: [C: 032] Memcached: tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/102698 (owner: 10Mark Bergsma) [15:36:15] !g 1234,5 [15:36:15] https://gerrit.wikimedia.org/r/#q,1234,5,n,z [15:40:55] (03CR) 10Bartosz Dziewoński: [C: 031] "This should be deployed today!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101005 (owner: 10Bartosz Dziewoński) [15:40:58] Reedy: ^ [15:45:47] (03PS1) 10Mark Bergsma: Memcached: Convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102700 [15:46:51] (03CR) 10jenkins-bot: [V: 04-1] Memcached: Convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102700 (owner: 10Mark Bergsma) [15:50:10] (03PS2) 10Mark Bergsma: Memcached: Convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102700 [15:51:41] (03CR) 10Mark Bergsma: [C: 032] Memcached: Convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/102700 (owner: 10Mark Bergsma) [15:54:51] (03CR) 10Odder: [C: 031] "Well done, especially with the 16px version which is much, much better than the one we're using now." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102586 (owner: 10RashiqAhmad) [16:17:05] yurik: ping [16:30:48] (03CR) 10Qgil: [C: 031] Updated community.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102586 (owner: 10RashiqAhmad) [16:34:22] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [16:34:22] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [16:34:53] (03PS2) 10Dan-nl: gwtoolset-runners [operations/puppet] - 10https://gerrit.wikimedia.org/r/102675 [16:40:30] (03CR) 10Dan-nl: "- added bug to commit message." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102675 (owner: 10Dan-nl) [16:41:18] (03PS2) 10Reedy: Undeploy ExpandTemplates extension [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 (owner: 10Dereckson) [16:41:23] (03CR) 10Reedy: [C: 032] Undeploy ExpandTemplates extension [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 (owner: 10Dereckson) [16:41:33] (03Merged) 10jenkins-bot: Undeploy ExpandTemplates extension [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97331 (owner: 10Dereckson) [16:41:37] wait wut MatmaRex, did you just +1d your own patch? ;-)) [16:43:32] twkozlowski: apparently :) [16:44:02] Reedy: is https://gerrit.wikimedia.org/r/101005 on your to-be-deployed-today list? :) [16:45:00] Not really [16:45:17] I'm tying to chase down the "and this" messages [16:45:49] (03PS3) 10Reedy: (bug 55630) $wgCategoryCollation = 'xx-uca-ckb' for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101005 (owner: 10Bartosz Dziewoński) [16:47:46] :/ that list should totally exist and be public somewhere [17:32:49] (03PS1) 10coren: Tool Labs: package installation round [operations/puppet] - 10https://gerrit.wikimedia.org/r/102714 [17:34:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:34:41] (03CR) 10coren: [C: 032] "Moar packages!!1!one" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102714 (owner: 10coren) [17:36:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:38:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:40:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:42:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:43:40] (03PS1) 10coren: Tool Labs: fix he locale [operations/puppet] - 10https://gerrit.wikimedia.org/r/102717 [17:44:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:45:44] (03CR) 10coren: [C: 032] "Baby fixlet." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102717 (owner: 10coren) [17:46:10] !log reedy synchronized php-1.23wmf8 'Staging' [17:46:27] Logged the message, Master [17:46:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:47:22] !log reedy updated /a/common to {{Gerrit|Ib6bc15109}}: previously missed deleted wikis, bug 58680 [17:47:30] (03PS1) 10Reedy: Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102718 [17:47:39] Logged the message, Master [17:47:47] (03PS2) 10Reedy: Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102718 [17:47:54] (03CR) 10Reedy: [C: 032] Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102718 (owner: 10Reedy) [17:48:05] (03Merged) 10jenkins-bot: Add/update symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102718 (owner: 10Reedy) [17:48:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:48:41] !log reedy synchronized docroot and w [17:48:58] Logged the message, Master [17:50:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:52:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:54:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:56:26] !log reedy started scap: Build l10n cache and testwiki to 1.23wmf8 [17:56:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [17:56:41] Logged the message, Master [17:58:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 05:30:46 PM UTC [18:00:17] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Thu Dec 19 18:00:12 UTC 2013 [18:02:37] PROBLEM - Puppet freshness on srv241 is CRITICAL: Last successful Puppet run was Thu 19 Dec 2013 06:00:12 PM UTC [18:03:22] (03PS1) 10Merlijn van Deen: Set default sql server for mysql [operations/puppet] - 10https://gerrit.wikimedia.org/r/102721 [18:09:00] (03CR) 10Dzahn: "hey, linting is good, no doubt about it and thanks, please just dont expect me to merge any LDAP changes on Friday before holidays:)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [18:11:40] (03CR) 10Dzahn: "also historically this would have been a classic Ryan Lane thing but i'm unsure if he still wants to look nowadays, understandably" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [18:14:34] (03CR) 10Matanya: "No horry or worry." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102629 (owner: 10Matanya) [18:21:49] hey Aaron|home, have you had a chance to look at https://bugzilla.wikimedia.org/show_bug.cgi?id=58692. we need to resolve this before we can properly test on production [18:21:57] (03PS1) 10Ottomata: Adding varnish_name parameter [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/102723 [18:22:20] (03CR) 10Ottomata: [C: 032 V: 032] Adding varnish_name parameter [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/102723 (owner: 10Ottomata) [18:25:21] (03PS1) 10Ottomata: Now logging frontend varnish logs through varnishkafka [operations/puppet] - 10https://gerrit.wikimedia.org/r/102727 [18:25:26] (03CR) 10Nikerabbit: Tool Labs: package installation round (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102714 (owner: 10coren) [18:26:46] (03CR) 10Ottomata: [C: 032 V: 032] Now logging frontend varnish logs through varnishkafka [operations/puppet] - 10https://gerrit.wikimedia.org/r/102727 (owner: 10Ottomata) [18:27:11] (03CR) 10coren: Tool Labs: package installation round (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102714 (owner: 10coren) [18:30:36] RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Thu Dec 19 18:30:28 UTC 2013 [18:31:45] (03PS1) 10Odder: Enable AbuseFilter block option on itwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102728 [18:32:08] !log reedy finished scap: Build l10n cache and testwiki to 1.23wmf8 [18:32:25] Logged the message, Master [18:33:31] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf7 till window [18:33:48] Logged the message, Master [18:36:42] (03CR) 10Aaron Schulz: [C: 04-1] gwtoolset-runners (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102675 (owner: 10Dan-nl) [18:41:02] Aaron|home, okay, i'm fine with abandoning that commit, but how do we resolve the issue? [18:41:36] huh? [18:41:44] I just meant go back to the prior patchset [18:43:29] right, i'll abandon that commit, but we're having an issue on the beta cluster [18:44:17] Aaron|home, hashar and i tried to detail it out here https://bugzilla.wikimedia.org/show_bug.cgi?id=58692 [18:46:05] (03Abandoned) 10Dan-nl: gwtoolset-runners [operations/puppet] - 10https://gerrit.wikimedia.org/r/102675 (owner: 10Dan-nl) [18:53:17] (03CR) 10Dzahn: "thanks for the positive review, Ori." [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn) [18:53:30] (03PS2) 10Reedy: Add modify restricted to sysop at Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101605 (owner: 10John F. Lewis) [18:53:36] (03CR) 10Reedy: [C: 032] Add modify restricted to sysop at Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101605 (owner: 10John F. Lewis) [18:53:57] mutante: JFDI! [18:54:28] (03Merged) 10jenkins-bot: Add modify restricted to sysop at Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101605 (owner: 10John F. Lewis) [18:54:31] ori_: any other time, but not Friday before major holiday and being on duty [18:54:41] ah, fair [18:54:43] too much on list [18:54:45] (03PS2) 10Reedy: Add Grants namespace to $wgNamespacesToBeSearchedDefault [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101995 (owner: 10Odder) [18:54:49] to get done before tonight [18:54:53] (03CR) 10Reedy: [C: 032] Add Grants namespace to $wgNamespacesToBeSearchedDefault [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101995 (owner: 10Odder) [18:55:33] (03Merged) 10jenkins-bot: Add Grants namespace to $wgNamespacesToBeSearchedDefault [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101995 (owner: 10Odder) [18:55:47] (03PS2) 10Reedy: Create a Portal namespace on the Odia Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101890 (owner: 10Odder) [18:55:52] (03CR) 10Reedy: [C: 032] Create a Portal namespace on the Odia Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101890 (owner: 10Odder) [18:56:05] (03Merged) 10jenkins-bot: Create a Portal namespace on the Odia Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101890 (owner: 10Odder) [18:57:27] (03PS2) 10Reedy: Enable AbuseFilter block option on itwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102728 (owner: 10Odder) [18:57:32] (03CR) 10Reedy: [C: 032] Enable AbuseFilter block option on itwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102728 (owner: 10Odder) [18:57:44] (03Merged) 10jenkins-bot: Enable AbuseFilter block option on itwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102728 (owner: 10Odder) [18:58:34] (03PS2) 10Reedy: Add an alias for NS_PROJECT on Bengali Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97206 (owner: 10Odder) [18:58:42] (03CR) 10Reedy: [C: 032] Add an alias for NS_PROJECT on Bengali Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97206 (owner: 10Odder) [18:58:47] and I wondered where did all the mail come from... [18:58:53] (03Merged) 10jenkins-bot: Add an alias for NS_PROJECT on Bengali Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97206 (owner: 10Odder) [18:59:11] (03PS2) 10Reedy: Add an alias to NS_PROJECT on Old English Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97376 (owner: 10Odder) [18:59:15] (03CR) 10Reedy: [C: 032] Add an alias to NS_PROJECT on Old English Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97376 (owner: 10Odder) [18:59:25] (03Merged) 10jenkins-bot: Add an alias to NS_PROJECT on Old English Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97376 (owner: 10Odder) [19:00:27] !log reedy updated /a/common to {{Gerrit|I22cfefa3c}}: Add an alias to NS_PROJECT on Old English Wiktionary [19:00:33] (03PS1) 10Reedy: All wikipedias to 1.23wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102734 [19:00:42] Logged the message, Master [19:00:55] (03CR) 10Dzahn: "removed myself from reviewers, fine with anything" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97190 (owner: 10Faidon Liambotis) [19:01:13] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikipedias to 1.23wmf7 [19:01:27] (03CR) 10Reedy: [C: 032] All wikipedias to 1.23wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102734 (owner: 10Reedy) [19:01:28] Logged the message, Master [19:01:33] (03Merged) 10jenkins-bot: All wikipedias to 1.23wmf7 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102734 (owner: 10Reedy) [19:01:45] (03PS2) 10Reedy: Enabling Persian Wikipedia Education Program [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99739 (owner: 10Ebrahim) [19:01:52] (03CR) 10Reedy: [C: 032] Enabling Persian Wikipedia Education Program [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99739 (owner: 10Ebrahim) [19:02:00] (03Merged) 10jenkins-bot: Enabling Persian Wikipedia Education Program [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99739 (owner: 10Ebrahim) [19:03:26] (03PS4) 10Reedy: (bug 55630) $wgCategoryCollation = 'xx-uca-ckb' for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101005 (owner: 10Bartosz Dziewoński) [19:03:32] (03CR) 10Reedy: [C: 032] (bug 55630) $wgCategoryCollation = 'xx-uca-ckb' for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101005 (owner: 10Bartosz Dziewoński) [19:03:41] (03Merged) 10jenkins-bot: (bug 55630) $wgCategoryCollation = 'xx-uca-ckb' for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101005 (owner: 10Bartosz Dziewoński) [19:03:59] (03CR) 10Dzahn: "it looks good, but definitely should have Labs architects on it. adding AndrewBogott and Coren" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97007 (owner: 10ArielGlenn) [19:04:13] (03PS3) 10Reedy: Rename Persian Wikibooks per community consensus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102482 (owner: 10Ebrahim) [19:04:22] (03CR) 10Reedy: [C: 032] Rename Persian Wikibooks per community consensus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102482 (owner: 10Ebrahim) [19:04:36] (03Merged) 10jenkins-bot: Rename Persian Wikibooks per community consensus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102482 (owner: 10Ebrahim) [19:05:18] !log reedy synchronized wmf-config/ [19:05:34] Logged the message, Master [19:05:48] Aaron|home, do the notes in the bug help clarify the issue at all? [19:06:56] !log reedy updated /a/common to {{Gerrit|I952672334}}: Rename Persian Wikibooks per community consensus [19:07:01] (03PS1) 10Reedy: phase1 wikis to 1.23wmf8 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102736 [19:07:12] Logged the message, Master [19:07:27] (03CR) 10Dzahn: [C: 04-1] "when i created this the status was like it's not there yet but coming very soon. we got a mail from legal back then. so i waited for confi" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/88705 (owner: 10Dzahn) [19:07:41] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: phase1 wikis to 1.23wmf8 [19:07:41] (03CR) 10coren: "This really needs to be tested extensively in a labs project; I'm a little worried about puppet ordering of class evaluation which is know" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97007 (owner: 10ArielGlenn) [19:07:58] Logged the message, Master [19:08:13] * Reedy wonders what else he can deploy [19:08:51] (03CR) 10Reedy: [C: 032] phase1 wikis to 1.23wmf8 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102736 (owner: 10Reedy) [19:08:58] Reedy: no more pending mini bugs in bugzilla where a tiny community waits for a fix of a typo in a namespace ?:) [19:09:00] (03Merged) 10jenkins-bot: phase1 wikis to 1.23wmf8 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102736 (owner: 10Reedy) [19:09:02] (03CR) 10Andrew Bogott: "I think that Ryan and Mike are aggressively refactoring these classes right now. So unless you're working in concert with them it might b" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97007 (owner: 10ArielGlenn) [19:09:35] I've not actually checked bugzilla [19:09:36] https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/mediawiki-config,n,z [19:09:49] (03PS4) 10Reedy: (bug 29902) Clean up InitialiseSettings.php, step 4/∞ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95284 (owner: 10Odder) [19:09:57] (03CR) 10Reedy: "Woah. It rebased!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95284 (owner: 10Odder) [19:10:13] (03CR) 10Reedy: [C: 032] (bug 29902) Clean up InitialiseSettings.php, step 4/∞ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95284 (owner: 10Odder) [19:10:22] (03Merged) 10jenkins-bot: (bug 29902) Clean up InitialiseSettings.php, step 4/∞ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95284 (owner: 10Odder) [19:10:52] (03PS11) 10Reedy: Updated internal.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 (owner: 10RashiqAhmad) [19:10:55] (03CR) 10Reedy: [C: 032] Updated internal.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 (owner: 10RashiqAhmad) [19:11:05] (03Merged) 10jenkins-bot: Updated internal.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102354 (owner: 10RashiqAhmad) [19:11:18] (03PS7) 10Reedy: Updated wikibooks.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102297 (owner: 10RashiqAhmad) [19:11:25] (03CR) 10Reedy: [C: 032] Updated wikibooks.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102297 (owner: 10RashiqAhmad) [19:11:37] (03Merged) 10jenkins-bot: Updated wikibooks.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102297 (owner: 10RashiqAhmad) [19:11:58] Reedy: and next i see you updating .icos:) nice [19:12:01] (03PS5) 10Reedy: Update favicon wiktionary/si.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100949 (owner: 10Tholam) [19:12:06] (03CR) 10Reedy: [C: 032] Update favicon wiktionary/si.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100949 (owner: 10Tholam) [19:12:18] (03Merged) 10jenkins-bot: Update favicon wiktionary/si.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100949 (owner: 10Tholam) [19:12:18] you are making Google program volunteers happy there i think [19:12:32] Numerous have been reviewed and ok'd, so no real reason for them to sit around [19:12:36] https://gerrit.wikimedia.org/r/#/c/95284/ I submitted that... [19:12:36] :) qgil loves you [19:12:40] had to be crazy. [19:12:58] (03PS4) 10Reedy: Updated community.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102586 (owner: 10RashiqAhmad) [19:13:02] (03CR) 10Reedy: [C: 032] Updated community.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102586 (owner: 10RashiqAhmad) [19:13:21] https://gerrit.wikimedia.org/r/#/c/99299/ Reedy? [19:13:37] yay, reedy wishlist hour, he wants moar [19:13:48] (03Merged) 10jenkins-bot: Updated community.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102586 (owner: 10RashiqAhmad) [19:14:24] Reedy: while on it, can you abandon https://gerrit.wikimedia.org/r/#/c/100326/ ? [19:14:35] you just merged a different patch that changed this fav [19:14:43] what is a step 4/∞ ? is that the infinity sign ? nice [19:14:52] (03PS2) 10Reedy: Preference "Email me when a page or file on my watchlist is changed" explicitly false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100958 (owner: 10Nemo bis) [19:14:58] (03CR) 10Reedy: [C: 032] Preference "Email me when a page or file on my watchlist is changed" explicitly false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100958 (owner: 10Nemo bis) [19:15:08] (03Merged) 10jenkins-bot: Preference "Email me when a page or file on my watchlist is changed" explicitly false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100958 (owner: 10Nemo bis) [19:16:02] mutante: it is, though it doesn't make any sense from a mathematical point of view, I'm told :) [19:16:42] well it depends on how you define / of course [19:16:50] (03CR) 10Ebrahim: "Thank you reedy, that is best offer for our 10th anniversary! :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/99739 (owner: 10Ebrahim) [19:17:46] PROBLEM - Host mw31 is DOWN: CRITICAL - Plugin timed out after 15 seconds [19:17:52] twkozlowski: that would be dividing 4 by infinity? [19:17:56] how much is that [19:17:59] 0? [19:18:36] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.44 ms [19:19:03] I think Nemo_bis studies mathematics :) [19:19:09] * Nemo_bis hides [19:19:20] http://answers.yahoo.com/question/index?qid=20080830092252AAxBbym [19:19:21] But a 0 as far as I can see, yes. [19:19:40] it says the answer is ε [19:19:47] !log reedy synchronized wmf-config/ [19:19:52] http://www.wolframalpha.com/input/?i=4%2F%E2%88%9E [19:19:54] 0! [19:19:59] * twkozlowski looks up ε on Wikipedia [19:20:04] Logged the message, Master [19:20:29] !log reedy synchronized docroot and w [19:20:45] Logged the message, Master [19:20:54] wolfram alpha rocks, yea [19:21:08] keep forgetting to use it more often [19:21:35] Reedy: https://gerrit.wikimedia.org/r/#/c/101889/ :-( [19:25:14] twkozlowski: The change could not be rebased due to a path conflict during merge. [19:25:18] twkozlowski: maybe you should replace ∞ by googol if you wanna be nerdy about it:) http://en.wikipedia.org/wiki/Large_numbers#Computers_and_computational_complexity [19:26:42] mutante: will try next time! [19:26:49] *g* [19:27:03] grrit-wm leaving aint so nice [19:27:55] and ottomata as well.. possibly combined with it [19:27:58] ah [19:28:21] ok, that looks like bot maintenance now [19:33:06] PROBLEM - NTP on mw31 is CRITICAL: NTP CRITICAL: Offset unknown [19:34:26] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [19:34:26] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [19:35:23] (03PS2) 10Odder: Add Malayalam aliases for NS_MODULE, NS_MODULE_TALK [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/101889 [19:36:36] (03PS1) 10Dan-nl: annotating-domain-whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102739 [19:38:06] RECOVERY - NTP on mw31 is OK: NTP OK: Offset 0.0003184080124 secs [19:42:19] Special:Version doesn't show extensions' GitInfo on production wikis, but does on beta-labs. Is that a bug? [19:43:11] Yes/no [19:43:14] And also known issue [19:43:16] greg-g, i want to get a few minor fixes in prod, as i'm not too happy with all those invalid message parameter warnings. When would be a good time? [19:43:17] https://gerrit.wikimedia.org/r/#/c/102601/ [19:43:31] * yurik wonders if Reedy wants to do it... [19:43:36] * greg-g looks [19:43:50] It's still my window! [19:43:52] greg-g, its for this one https://gerrit.wikimedia.org/r/#/c/102604/ [19:44:14] Where/what is using Bug58676 as a log group? [19:44:23] Reedy, ^^ [19:44:27] Ah [19:44:42] * greg-g nods [19:44:55] * yurik doesn't like mess in fatal monitor [19:45:02] Reedy, want to push it out? :) [19:45:03] yurik: grep -v [19:45:15] grep -v 'Search backend error' | grep -v -i 'swift' [19:45:16] :D [19:45:41] Reedy, grep -v only helps with the recursive lang context - because its not a core issue and anomie is working on it :) [19:45:56] wrong message props should be dealt with :) [19:52:30] (03PS3) 10Reedy: Add logging for "Invalid message parameter" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102601 (owner: 10Yurik) [19:52:35] (03CR) 10Reedy: [C: 032] Add logging for "Invalid message parameter" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102601 (owner: 10Yurik) [19:52:42] !log reedy synchronized php-1.23wmf7/includes/Message.php [19:52:55] thanks Reedy !!!! [19:52:57] :) [19:52:59] Logged the message, Master [19:53:42] !log reedy synchronized wmf-config/InitialiseSettings.php [19:53:58] Logged the message, Master [19:54:02] (03Merged) 10jenkins-bot: Add logging for "Invalid message parameter" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102601 (owner: 10Yurik) [19:55:31] (03PS1) 10Yurik: AAA!!! Regex BUG in ZERO. Need depl right away. [operations/puppet] - 10https://gerrit.wikimedia.org/r/102742 [19:55:34] paravoid, it seems like a found a massive production bug :(( ^ [19:55:49] i know how much you like those :) [19:56:17] i'm surprised varnish did not complain about invalid regexes [19:56:53] awesome [19:57:36] or who else has +2 powers for puppets [19:58:22] ori [19:58:29] ori_, ^^ [19:58:36] urgently needed :( [19:58:51] mutante is on duty... [19:59:09] ACHTUNG MUTANTE [19:59:24] if you have a problem, and solve it with regex, you have two problems :( [19:59:25] uh [19:59:32] catching up [19:59:39] ori_, https://gerrit.wikimedia.org/r/#/c/102742/1/templates/varnish/zero.inc.vcl.erb [20:00:46] someone from ops should merge that [20:00:46] Aaron|home, do the notes in the bug, https://bugzilla.wikimedia.org/show_bug.cgi?id=58692, help clarify the issue at all? [20:00:52] mutante, are you around? [20:01:17] what is breaking at the moment? [20:01:22] oh yes! [20:01:28] yurik, I can merge. Confident? [20:01:38] It looks right to me... [20:01:39] basically all of zero are not identifying as such [20:01:43] go for it [20:01:51] unless i am totally missing something, doubtful [20:02:04] andrewbogott, yes, please [20:02:08] ugh, i;m here [20:02:14] but if others, please do just as well [20:02:29] i was reading on varnish for bd808 :p [20:02:40] (03CR) 10Andrew Bogott: [C: 032] AAA!!! Regex BUG in ZERO. Need depl right away. [operations/puppet] - 10https://gerrit.wikimedia.org/r/102742 (owner: 10Yurik) [20:02:45] thx [20:02:46] (03PS1) 10Ori.livneh: Configure Varnish not to cache scholarship app reqs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102744 [20:02:46] thanks! [20:02:50] ^ bd808 [20:03:13] varnish, the program we all love... to hate... to love... undecided... :) [20:03:15] yurik, will take a few minutes as puppet rolls out the change. [20:03:20] yep [20:03:22] thx! [20:03:26] ori_: I suppose that's the most direct way to do it :) [20:04:17] andrewbogott: can you merge 102744 as well? i think bd808 and i will ask paravoid if that's the best way to do it, but it is adequate for the moment [20:04:27] the scholarship app is very (very, very) low traffic [20:04:38] i can do the puppet-merge and sync on the varnishes [20:04:51] but a sanity check from someone in ops would be good [20:05:13] you can see that we're already return(pass)-ing a few lines below if the req is not GET or HEAD [20:05:20] I'm in no rush but would like to find a solution [20:06:34] ori_: I can merge if you like but I have definitely never read a varnish config :) [20:06:55] andrewbogott: if you like you can +1 it and i'll +2 so it's clear who's on the hook if anything breaks. but nothing will break. [20:07:00] 'k [20:07:33] (03CR) 10Andrew Bogott: [C: 031] "seems safe." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102744 (owner: 10Ori.livneh) [20:07:42] ori_: are you the king of graphite? [20:07:55] i'm not sure how to answer that [20:07:55] I ask regarding https://bugzilla.wikimedia.org/show_bug.cgi?id=54713 [20:08:02] let me look, just a sec [20:08:05] no rush [20:09:06] (03CR) 10Ori.livneh: [C: 032] "@faidon: merging this for now, but -- is this the best way to handle it? feedback plz" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102744 (owner: 10Ori.livneh) [20:12:52] andrewbogott: i suffer from that bug too :) [20:12:58] (03PS1) 10Ori.livneh: Fix whitespace added in Ie5c2e260a [operations/puppet] - 10https://gerrit.wikimedia.org/r/102746 [20:14:39] (03CR) 10BryanDavis: "An alternate way to handle it would be to have the app send the proper response headers as proposed in I499d68e. I'm open to either soluti" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102744 (owner: 10Ori.livneh) [20:15:36] (03CR) 10Ori.livneh: [C: 032 V: 032] Fix whitespace added in Ie5c2e260a [operations/puppet] - 10https://gerrit.wikimedia.org/r/102746 (owner: 10Ori.livneh) [20:15:58] ottomata: hey! how are those building coming along? [20:17:25] oh haven't checked! [20:17:26] let's see [20:18:26] i think they all completed successfully [20:18:31] the last (proper) output I have is [20:18:34] 184 jobs left on the queue. [20:18:52] but, while read loop died because I had that pesky blank line [20:18:54] removed now. [20:19:09] andrewbogott: replied on the bug [20:19:15] thanks [20:19:51] manybubbles: is there anytihng else I should check? [20:19:53] or should I do the second pass [20:20:35] ottomata: nah, that should be fine. second pass now! [20:22:37] Aaron|home: is there someone else who might be able to help with https://bugzilla.wikimedia.org/show_bug.cgi?id=58692 ? [20:38:39] (03PS1) 10Jeremyb: 2 new education redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/102753 [20:38:42] (03CR) 10jenkins-bot: [V: 04-1] 2 new education redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/102753 (owner: 10Jeremyb) [20:39:59] i ran some stats on apache log, seems like this is the most common one: 439033 File does not exist: /usr/local/apache/common/docroot/wikipedia.org/upload.wikimedia.org [20:40:17] followed by 199477 File does not exist: /usr/local/apache/common/docroot/wikipedia.org/materia [20:40:23] 119635 File does not exist: /usr/local/apache/common/docroot/wikipedia.org/Special:Booksources [20:40:37] is that normal? [20:42:26] greg-g: we have a fix for bug 58705, will require deploying Echo and Flow patches. Do you want it now or in an LD? [20:43:05] (03PS2) 10Jeremyb: 2 new education redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/102753 [20:43:33] (03CR) 10jenkins-bot: [V: 04-1] 2 new education redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/102753 (owner: 10Jeremyb) [20:44:16] something wrong with jenkins? hashar can you look? ^^ [20:44:20] * jeremyb doesn't have time right now [20:48:02] jeremyb: It's a lint failure in wwwportals.conf caused by mod_rewrite not being loaded? [20:48:21] bd808: i saw something like that. but that's not related to my change... [20:48:31] * bd808 thinks that sounds sort of broken [20:48:37] next step would be to look at when it started failing. but i can't dig now [20:49:08] * jeremyb runs away [20:49:21] yurik: People send garbage requests all the time [20:49:38] First failure from https://gerrit.wikimedia.org/r/#/c/101220/ [20:49:43] RoanKattouw, yes, but half a million? [20:50:12] * bd808 blames puppet [20:50:16] Yeah don't know why that is [20:51:06] I blame relative URLs. Perhaps. [20:51:25] and templates, that's for sure :-) [20:55:26] hey [20:55:51] bd808: jeremyb yup apache-config tests are broken :/ [20:57:33] spagewmf: how about at 1:30ish? [21:08:03] hi mutante. [21:08:06] anything I can do to help? [21:08:50] !log csteipp started scap: bug 58472, 57550, 58553 [21:09:06] Logged the message, Master [21:23:52] (03PS1) 10Ori.livneh: Add user-agent header to the format spec of EventLogging's varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 [21:24:36] mutante: Tim's redirects thing has merged. i thought about it and then forgot to use it which is why i made a PS2. (PS2 is just from running Tim's script) [21:27:08] (03PS2) 10Ori.livneh: Add user-agent header to the format spec of EventLogging's varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 [21:27:26] mutante: also, it's thursday! AFAIK. :-) [21:30:13] hashar: what do you think about a jenkins job that rejects the change if the redirects.conf is not the same as what is generated from the current redirects.dat? [21:31:44] jeremyb: I can't look at it and on vacations tomorrow evenign [21:32:05] jeremyb: so I am afraid it is going to stay broken for a few days unless someone look at the test script [21:32:14] hashar: can't look at it ever? and i asked what you think not to do it :) [21:32:22] the job probably need to have a command added to it that would expand the .conf files [21:34:20] (03PS3) 10Nemo bis: Add user-agent header to the format spec of EventLogging's varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [21:41:55] jeremyb: basically I don't think about it :( [21:42:03] jeremyb: not enough time to context switch to it [21:42:37] jeremyb: yep, i saw, i just havent done an actual redirect since it's there [21:42:51] looks all cool and better than before though [21:43:05] greg-g: I don't believe enough of the wikis are built at this point to warrant turning them on as a betafeature. it'd be just one or two, more trouble then it is worth unfortunately [21:43:11] jeremyb: lol about thursday [21:43:22] jeremyb: i seriously didnt mean to lie about the day :p [21:43:34] i think i said Friday before today here [21:44:02] manybubbles: /me nods [21:47:21] !log csteipp finished scap: bug 58472, 57550, 58553 [21:47:39] Logged the message, Master [21:49:23] Tue Feb 12 00:50:07 2013 Probably want to poke Antoine and Brad about this.. [21:49:30] #4510: mctest.php returns tampa hosts [21:49:46] :D [21:50:41] (03CR) 10Aquifacae: "Hello Mr. Gil," [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102602 (owner: 10Aquifacae) [21:50:50] (03Abandoned) 10Aquifacae: Made proper larger favicon wiktionary/en.ico. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102602 (owner: 10Aquifacae) [21:51:05] Reedy: #Will file a bugzilla ticket for this# .. Did you? [21:51:13] then i just link it and thats it for now [21:51:32] oh, you did, wait [21:51:40] https://bugzilla.wikimedia.org/show_bug.cgi?id=44889 [21:52:10] can i say having BZ is enough and close RT? [21:52:59] i will, reopen is cheap [21:53:04] mutante: ? [21:53:25] anomie: it's a really old ticket in ops requests [21:53:31] and its identical to the bugzilla above [21:54:13] Reedy pasted this back then [21:54:15] # Cluster-dependent files for database and memcached [21:54:15] require( getRealmSpecificFilename( "$wmfConfigDir/db.php" ) ); [21:54:15] require( getRealmSpecificFilename( "$wmfConfigDir/mc.php" ) ); [21:54:24] followed by that sentence mentioning you [21:55:07] spagewmf: whenver you're ready [22:03:24] (03PS1) 10Yurik: Updated ZERO URL to get just the carrier IPs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102824 [22:03:47] bblack, ^ [22:06:09] (03CR) 10DarTar: [C: 031] "/me looking for the GerritLove button. Thanks for getting this started." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [22:08:30] !es [22:08:31] external store (and sometimes elasticsearch, but we try not to) [22:08:43] yay two definitions [22:08:52] thats why i added it [22:08:53] heh [22:09:10] eh, or demon did [22:09:15] we added several that day [22:09:24] !cp [22:09:25] caching proxy (squid or varnish) [22:10:20] (03CR) 10BBlack: [C: 032 V: 032] Updated ZERO URL to get just the carrier IPs [operations/puppet] - 10https://gerrit.wikimedia.org/r/102824 (owner: 10Yurik) [22:11:03] yurik: ^ [22:11:23] thx! [22:15:06] (03CR) 10Ottomata: Add user-agent header to the format spec of EventLogging's varnishncsa instance (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [22:15:30] (03PS3) 10Yurik: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 [22:19:07] greg-g, it's going to be a while have to +2 to approve and backport the fixes, and we can't repro on our labs machine [22:20:23] anomie: regarding bug 58705, is OAuth set up on beta-labs ? [22:20:34] spagewmf: gotcha, let me know what you need/want to do [22:20:35] spagewmf: Should be [22:35:26] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC [22:35:26] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC [22:58:58] (03CR) 10Ori.livneh: Add user-agent header to the format spec of EventLogging's varnishncsa instance (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [23:17:39] manybubbles: btw, the second pass finished