[00:18:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.424 second response time [00:28:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [01:03:32] New review: Krinkle; "The one time you bypass Jenkins.." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52315 [01:28:19] New patchset: Krinkle; "Rename legacy 'live-1.5' to 'mw'." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53125 [01:34:56] New patchset: Krinkle; "Introduce wmgUseVectorFooterCleanup" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53126 [01:34:56] New patchset: Krinkle; "$wmgUseVectorFooterCleanup = true; for commonswiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53127 [01:42:51] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53126 [01:43:17] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53127 [01:51:02] New patchset: Aklapper; "Add query and formatting for most urgent issues in Bugzilla" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53128 [01:51:27] Change abandoned: Aklapper; "Superseded by https://gerrit.wikimedia.org/r/#/c/53128" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52401 [02:29:28] !log LocalisationUpdate completed (1.21wmf11) at Mon Mar 11 02:29:27 UTC 2013 [02:29:33] Logged the message, Master [02:41:00] !log krinkle synchronized wmf-config/InitialiseSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d' [02:41:06] Logged the message, Master [02:42:58] !log krinkle synchronized wmf-config/CommonSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d' [02:43:04] Logged the message, Master [02:52:06] !log LocalisationUpdate completed (1.21wmf10) at Mon Mar 11 02:52:06 UTC 2013 [02:52:12] Logged the message, Master [03:53:35] * jeremyb_ wondres if someone feels like a deploy :P [03:53:48] too bad i didn't notice sooner, krin kle just did a deploy [03:54:07] (not an emergency... but writing a commit now) [03:55:21] and of course i go to the wrong repo... [03:55:35] * Damianz gives jeremyb_ a cookie [03:55:48] i hope it's good [03:57:12] hrm, duplicate tfincs [04:11:49] New review: Jeremyb; "This broke some URLs like https://wikimania2006.wikimedia.org/wiki/Proceedings:LP1 (which now appear..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/51288 [04:12:15] New patchset: Jeremyb; "fix wikimania pages like [[wm2006:Proceedings:LP1]]" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53133 [04:12:47] wonders* [04:15:14] New patchset: Jeremyb; "fix wikimania pages like [[wm2006:Proceedings:LP1]]" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53133 [04:15:52] legoktm: ^^ is what my TS query would have been about [04:16:25] ah [04:57:26] New review: Tim Starling; "mw? Why not name it after what is actually in it, rather than what those wrapper scripts point to? T..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/53125 [05:00:14] huh, hashar's working but not here :/ [05:00:23] New review: GreenReaper; "Looks good. FWIW, the namespace seems to have fallen into disuse as of 2010 (though it's still prese..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/53133 [05:01:00] welp, James_F has a review req from me :) (see last review from GreenReaper above) [05:01:22] jeremyb_: Yeah, we don't seem to use it any more, but it doesn't harm. [05:01:27] jeremyb_: Want me to +1? [05:01:53] James_F: well it broke old URLs. he came to the channel complaining about it. (and also it made links into redlinks) [05:02:04] * James_F nods. [05:02:18] New review: Jforrester; "Seems fine from my POV. Even if we don't use the namespace, we should make sure it still works. :-)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/53133 [05:02:24] * James_F bows. [05:02:42] danke :) [05:03:03] 09 23:22:57 < GreenAway> Hey, what happened to the Wikimania 2006 website? I was linking to my proceedings page, and it's disappeared: http://wikimania2006.wikimedia.org/wiki/Proceedings:LP1 [05:03:07] 09 23:23:22 < GreenReaper> while e.g. http://wikimania2006.wikimedia.org/wiki/Presenters/Laurence_Parry is still there [05:03:10] 09 23:50:59 * GreenReaper imagines there is some kind of namespace-related issue. [05:03:41] now we can just let it sit for someone to deploy [05:03:44] * jeremyb_ sleeps [08:09:34] New review: Krinkle; "@Tim: Interesting point you bring up. I guess it depends on how you look at the directory. In the cu..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53125 [08:21:39] New review: Hashar; "Fixed bug 44134" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/46886 [09:52:52] New patchset: Petrb; "puppetized motd of bots project of labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53145 [10:04:23] Damianz can you review it ^ [10:04:28] I am puppet newb [13:22:25] New patchset: Mark Bergsma; "Move misc::monitoring to where it belongs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53157 [13:22:26] New patchset: Mark Bergsma; "Fully qualify global variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53158 [13:22:47] I'm baack [13:24:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52988 [13:25:19] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53157 [13:26:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53158 [13:30:58] New patchset: Mark Bergsma; "Run gmond as user 'ganglia' instead of 'nobody'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53159 [13:31:51] New patchset: Mark Bergsma; "Fully qualify global variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53160 [13:34:40] the ganglia manifests make me wanna stab my eyes out [13:34:53] ohhi [13:34:56] hi [13:36:37] ohhiiiii! [13:43:49] does anyone know how I push a new branch to gerrit for review? [13:45:54] ottomata: I'm the ultimate not-expert, but I can probably help. [13:46:23] ottomata: You mean you want to submit a changeset for review, right? [13:47:04] ottomata: First step: make sure you have git-review installed. :-) [13:47:07] yeah [13:47:11] i've pushed lots of stuff for review before [13:47:18] i'm trying to do something non standard for sure [13:47:45] i'm trying to maintain git-buildpackage branches in gerrit [13:47:49] i want to push the debian branch for review [13:48:08] the debian branch doesn't exist on the remote yet [13:48:14] Ah, you're probably way over /my/ head then. I've a black belt is svn-fu, but my git-fu is weak. :-) [13:48:35] ayeyy, ok, thanks anyway :) [13:48:47] yeah, currently, when I try to use git review to push a new branch [13:48:49] fatal: ambiguous argument 'remotes/gerrit/deb': unknown revision or path not in the working tree. [13:50:00] ottomata: create it in gerrit [13:50:10] New patchset: Mark Bergsma; "Fully qualify global variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53160 [13:50:11] New patchset: Mark Bergsma; "Run gmond as user 'ganglia' instead of 'nobody'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53159 [13:50:11] also [13:50:11] New patchset: Mark Bergsma; "Bug fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53163 [13:50:15] did you setup europium? [13:51:53] no not yet, that's going to take some time i think. the udp2log stuff is puppetized, but Jeff_Green has a bunch of fundraising stuff there that is not [13:52:13] also, i wanted to ask you guys about using oxygen's multicast stream rather than the hardcoded IPs in the frontend cache configs [13:52:35] can you perhaps silence the cron job while it's taking some time then? :) [13:52:35] we can set up udp2log on europium to consume from the multicast stream emitted from oxygen [13:52:42] cron job....>>.... [13:52:53] HM [13:52:53] sure [13:52:56] thanks [13:53:26] mark, how do I create a new branch in gerrit without an initial revision? [13:53:34] just create it [13:53:37] it'll work [13:54:45] gerrit wants an initial revision to start the branch from [13:55:07] pick the upstream branch you branched from [13:55:08] my branch did not start from a revision that was previously in gerrit (git-buildpackage created them all locally) [13:55:33] hm [13:55:38] if you have rights you can just force push iirc [13:55:54] yeah i can push, but i want to use git-review so paravoid can review the debian/ stuff [13:56:07] i'm doing something really non standard here, I think if I could delete gerrit repos then I think I could make this work [13:56:18] i really need to delete the existing one and then start over [13:58:13] and I guess refs/for/ doesn't work hm [13:58:22] New patchset: Ottomata; "Unpuppetizing udp2log and fundraising stuff from europium until Jeff and I get coordinated." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53164 [13:58:55] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53164 [14:00:51] (cron job SILENCED!) [14:01:22] Ahh! is ^demon is here to rescue me!? [14:01:30] (good morning :) ) [14:01:38] are you back in VA or still SF? [14:01:55] <^demon> Back in VA, I'm afraid :( [14:03:02] how does ganglia get this fucke dup [14:03:52] I'm in DC right now [14:04:23] ^demon, I gots a gerrit q, want me to plague you with it now or wait til you get settled? :p [14:04:41] <^demon> Shoot. [14:05:09] i'm trying to use git-buildpackage to build a kafka .deb, paravoid wants to review [14:05:12] using gerrit [14:05:22] git-buildpackage (at minimum) maintains 3 branches [14:05:42] before I started using git-buildpackage, I had previously created operations/debs/kafka [14:06:09] i think if we delete operations/debs/kafka, I can push the branches that don't need review directly, and then git-review master [14:06:12] but, more generally [14:06:15] if it isn't deleted [14:06:25] i can use a different branch name instead of 'master' for the debian-branch [14:06:28] but [14:06:36] how do I push a branch for review that isn't yet on gerrit [14:06:44] and that was created locally but not from one of the existing remote branches [14:06:44] ? [14:07:21] <^demon> Provided you've got direct pushing permissions on that branch, you'd just `git push origin [branchname]` [14:07:37] i want to push for review into gerrit [14:07:47] <^demon> Oh, for review. [14:07:49] <^demon> Derp. [14:08:06] :) [14:08:22] <^demon> `git push origin [branchname]:refs/for/[branchname]` [14:08:26] <^demon> I hate git-review :) [14:08:32] ok... [14:09:09] ! [remote rejected] deb -> refs/for/deb (branch deb not found) [14:09:19] the branch does not exist on the remote yet [14:09:33] and, gerrit won't let me create it there, unless I specify a revision to branch from [14:09:56] and I don't think I should, since the branch was not created from a revision that is in the remote (which is probably not good, and why we should delete this gerrit repo altogether :p ) [14:10:04] <^demon> argh... [14:10:29] <^demon> We could delete easily if https://gerrit.wikimedia.org/r/#/c/50385/ didn't exist yet (then it's just a matter from removing it from disk and flushing caches) [14:10:29] this really is a whole new repository [14:10:40] <^demon> Cleaning up comments & changes is annoying...hence the plugin. [14:10:49] hm, aye [14:11:19] well, in the past, you've just reverted to the empty first commit, right? [14:11:41] I *might* be able to make that work, because maybe then I can take my new repo and set the remote and then just do the regular push/review stuff? [14:11:43] without conflicts? [14:11:44] maybe? [14:11:48] <^demon> Lemme try something. [14:11:50] <^demon> Sec. [14:11:51] k [14:12:21] oh [14:12:24] ^demon [14:12:25] no [14:12:26] that's good [14:12:29] don't delete that one! [14:12:31] that's the puppet module [14:12:33] i'm working with the deb [14:12:39] https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/kafka [14:12:44] i don't think it has any gerrit patchsets [14:13:32] <^demon> Orly. [14:13:36] <^demon> Yay. [14:13:40] <^demon> We could just delete it. [14:13:47] dooooit [14:13:53] that's fine [14:13:59] i want a brand new repo there [14:14:12] i'm going to push 3 branches, 2 directly and master for review [14:17:27] <^demon> Repo deleted, should be able to recreate now. [14:17:36] New patchset: Mark Bergsma; "Fully qualify global variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53160 [14:17:36] New patchset: Mark Bergsma; "Bug fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53163 [14:17:36] New patchset: Mark Bergsma; "Run gmond as user 'ganglia' instead of 'nobody'" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53159 [14:17:37] New patchset: Mark Bergsma; "$ganglia_aggregator is not top level" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53166 [14:18:05] hokaayy, thanks ^demon, lets see if I don't mess this up again :) [14:18:49] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53166 [14:21:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53159 [14:21:40] hmm, ^demon [14:21:49] i shoudln't Create initial empty commit, right? [14:21:56] since I already have a repo locally? [14:22:02] <^demon> Right. [14:22:06] and, if so, will I run into the same problem when try to push for review? [14:22:10] since the branch won't be on the remote yet? [14:22:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53160 [14:22:56] <^demon> Should be possible...I've done it before. [14:23:01] ok [14:23:03] so i'll do that [14:23:07] then setup git review [14:23:15] then just git-review master [14:23:19] and see what happens :) [14:23:35] hmm, how does gerrit care about master? [14:23:39] i'd rather not call this master, [14:23:44] i'd like to call it 'deb' or 'debian' [14:23:45] New patchset: Mark Bergsma; "Fix syntax" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53167 [14:23:47] and not have a 'master' [14:23:51] will gerrit care? [14:24:50] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53163 [14:25:14] ^^demon? [14:25:33] <^demon> You can not have a master if you want... [14:25:42] <^demon> People will hate you for it though. Just ask Ryan ;-) [14:26:45] oh? well, I can do it, it just seems confusing to me since this repo is not for kafka source, only for building debs. buuut, i'll leave the master in, since that is the git-buildpackage default debian-branch [14:26:46] seems weird, but i'll leave it [14:26:51] New patchset: Mark Bergsma; "Fix syntax" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53167 [14:27:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53167 [14:28:35] no point in having master in those repos no [14:28:57] i will hate you for having master there [14:28:59] you agree mark? master is weird? [14:29:00] hahah [14:29:02] ok [14:29:07] it's confusing [14:29:12] good I agree [14:29:15] there's master but it isn't the master [14:29:30] ryan doesn't build debs, so he should stfu about this ;) [14:29:36] i'd call this branch 'debian', except that there is a directory named 'debian/', and git-review seems to have a bug when the branch is named the same as a file [14:29:44] er [14:29:45] do call it debian [14:29:48] please, stick to the standard [14:29:50] can I call it 'deb' [14:29:50] ? [14:29:58] oh, didn't know it was a standard [14:30:00] if git-review bugs on it then don't use git-review or fix it [14:30:10] git-buildpackage uses 'master' as the default debian-branch [14:30:22] i don't think so [14:30:32] i thought it was debian? [14:30:40] anyway, it it does use master then by all means, use master [14:30:49] not something weird made up [14:31:14] The debian-branch (the default branch name used in the Git repository is master) holds your current development work. That's the branch you usually cut your releases from and the default branch new upstream releases are merged onto. [14:31:47] am I reading that wrong, or is master what I should use then? [14:31:56] yes [14:31:57] ok [14:32:00] so master it is then [14:32:04] ok cool [14:33:20] hmm, ^demon, one more question [14:33:38] is it possible for me to push the first commit in this branch directly, and then the most recent (there are only 2), for review? [14:33:52] ahh, i can probably figure that out with google [14:38:11] <^demon> Yes, just have to adjust the acl on the repo temporarily to allow direct pushing. [14:38:25] i think it will let me push the first commit [14:39:12] just not sure how to push a single commit, trying: [14:39:16] git push origin c0c66f7d6a4eab0027418267b43eff9b7d1f9e36:master [14:39:20] error: unable to push to unqualified destination: master [14:39:20] The destination refspec neither matches an existing ref on the remote nor [14:39:20] begins with refs/, and we are unable to guess a prefix based on the source ref. [14:39:20] error: failed to push some refs to 'ssh://otto@gerrit.wikimedia.org:29418/operations/debs/kafka.git' [14:41:55] <^demon> Try c0c66f7d6a4eab0027418267b43eff9b7d1f9e36:refs/heads/master? [14:42:19] OOOO [14:42:39] yes! [14:42:40] that did it [14:42:56] New patchset: Ottomata; "Initial debian packaging using git-buildpackage" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/53170 [14:43:11] YES! [14:43:34] New patchset: Ottomata; "Initial debian packaging using git-buildpackage" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/53170 [14:52:21] YAYYYY [14:52:24] paravoid [14:52:24] https://gerrit.wikimedia.org/r/#/c/53170/ [14:52:50] thanks ^demon! [14:53:04] <^demon> yw. [14:53:25] New patchset: Ottomata; "Initial debian packaging using git-buildpackage" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/53170 [15:31:29] New patchset: Demon; "Don't keep any archived gerrit sshd logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53173 [15:43:39] New patchset: Alex Monk; "(bug 45952) Reenable TimedText namespace on foundationwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53174 [15:57:11] !log deployed new webstatscollector filter on locke. This replaces spaces in urls. [15:57:16] Logged the message, Master [16:19:03] New patchset: Jeremyb; "Don't keep any archived gerrit sshd logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53173 [16:19:08] ^demon: ^ [16:20:18] <^demon> Oh whoops, thanks for that. [16:27:41] ^demon: how do the logs get there at all? [16:27:52] i.e. what does the rotation [16:28:05] <^demon> gerrit does it. [16:41:40] <^demon> "Cannot run a MediaWiki script as a user in the group wikidev" :( [16:43:05] idk what to think about this ascii art from gitblit :P [16:43:20] <^demon> Heh, where? [16:43:38] /var/lib/gerrit2/review_site/logs/error_log of course! :) [16:43:46] ^demon: seen rt 4196 ? [16:43:48] <^demon> Oh, on startup. [16:44:21] <^demon> jeremyb_: No, I hadn't. [16:44:35] seems possibly relevant [16:44:42] although i'm not fully awake [16:44:58] <^demon> I was trying foreachwiki on fenari. [16:45:56] oh [16:46:04] was that an error msg from a script refusing to run? [16:46:39] <^demon> Shouldn't be...script worked fine just running it for a single wiki with mwscript [16:47:33] * jeremyb_ votes for asking Reedy [16:48:27] !log aaron synchronized php-1.21wmf11/includes/limit.sh 'deployed e77397fc1de324dfc6d7b515e9440bd564311dd9' [16:48:34] Logged the message, Master [16:48:46] notpeter: you around? [16:48:54] <^demon> jeremyb_: Or AaronSchulz :) [16:49:43] New review: Jeremyb; "See also http://www.w3.org/Provider/Style/URI.html" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53133 [16:49:47] <^demon> mwscript is his faul^H^H^H^H invention :) [16:50:19] * jeremyb_ knew not that history [16:53:33] New review: Daniel Kinzler; "(3 comments)" [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/47088 [16:54:17] New review: Daniel Kinzler; "uh... sorry, disregard that :)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/47088 [16:56:33] New review: Aude; "can someone explain how would it (not) work with something like de.wikidata.org/wiki/Wikidata:Forum ?" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/47088 [16:59:28] ^demon: Tim changed stuff [16:59:32] Some need running as apache and stuff.. [17:00:23] <^demon> i wanna run my script :\ [17:00:35] Reedy: https://gerrit.wikimedia.org/r/53133 [17:02:08] Which script? [17:02:49] <^demon> extensions/WikimediaMaintenance/cleanupSkinPrefs.php [17:02:57] <^demon> Runs fine by itself with mwscript. [17:03:01] <^demon> For a single wiki [17:03:05] <^demon> But I want foreachwiki [17:03:34] New patchset: Aaron Schulz; "Fixed typo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53181 [17:03:57] sudo -u apache foreachwiki extensions/WikimediaMaintenance/cleanupSkinPrefs.php [17:04:04] aawiki [17:04:04] ----------------------------------------------------------------- [17:04:04] aawiki: 4 users with old integer-style skin preferences [17:04:04] aawiki: 851 users with bogus skin properties [17:04:23] * Reedy leaves running [17:04:37] Does it need a --fix or something? [17:04:40] Reedy: yup, or use mwscriptwikiset [17:05:13] if( !$countOnly && $count > 0 ) { [17:05:19] ^ $count is undefined [17:05:53] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53181 [17:06:54] Change abandoned: Aude; "this seems to be resolved elsewhere now" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [17:07:27] New patchset: Reedy; "fix wikimania pages like [[wm2006:Proceedings:LP1]]" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53133 [17:07:33] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53133 [17:08:38] RobH: [17:09:09] does someone feel like updating interwiki map or should i file a bug? wikitech should get it's new URL [17:09:14] Change restored: Aude; "seems the change was never made." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [17:09:32] * aude has no idea if https://gerrit.wikimedia.org/r/#/c/49069/ is still needed for wikidata [17:09:52] its* [17:09:53] !log taking down search23 for MB replacement: https://rt.wikimedia.org/Ticket/Display.html?id=3423 [17:09:58] Logged the message, Master [17:10:03] sbernardin: ? [17:10:10] <^demon> Reedy: --count.... [17:10:17] <^demon> And change has to be pulled in first... [17:10:27] <^demon> (it's not live on 1.21wmf10, only 11) [17:10:55] <^demon> We only want to count for now. [17:14:15] sbernardin: checking to see if its in a pool iwth other servers [17:16:14] RobH: OK [17:16:19] grr [17:16:22] search is horrible. [17:16:49] ok, search 21, 22, 23, 26 are in same pool [17:16:55] so going to set 23 to false [17:17:09] !log setting search23 to false for mb swap [17:17:15] Logged the message, RobH [17:17:50] !log search23 out of search pool1, other servers in pool online [17:17:55] Logged the message, RobH [17:17:59] ^demon: without count it doesn't seem to do anything [17:18:09] sbernardin: as soon as it depools i will shut it down [17:18:12] Every time I run it: [17:18:13] aawiki: 4 users with old integer-style skin preferences [17:18:13] aawiki: 851 users with bogus skin properties [17:18:28] <^demon> Reedy: Because you need my patch pulled in first.... [17:18:33] <^demon> As I said. [17:18:53] !log search23 shutting down for mb sap per rt4323, has been disabled in pybal [17:18:55] You didn't say that until after I started running it [17:18:58] Logged the message, RobH [17:18:58] No harm done then [17:19:03] <^demon> Well, I didn't say to run it ;-) [17:19:10] Are you going to run it yourself now you know how? :p [17:19:15] sbernardin: once it shuts off it is all yours [17:19:38] once it comes back online it will prolly need reinstall, so lemme konw when its fixed please =] [17:19:52] <^demon> Yessss. [17:20:24] New patchset: Alex Monk; "(bug 43524) Restrict upload on itwikivoyage to sysops" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53183 [17:20:34] OK [17:21:25] <^demon> After I finish my lunch [17:22:49] its not lunch time! [17:22:59] oh yes, poor east coaster [17:23:04] * RobH is being a shit ;] [17:23:18] Lol [17:23:35] sbernardin: so ^demon is moving out here, but not until summer [17:23:38] so i have to give him hell. [17:23:52] I see [17:26:03] <^demon> July 10th \o/ [17:26:07] New review: Michał Łazowik; "(3 comments)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/47088 [17:27:20] * AaronSchulz wtfs at http://en.wikipedia.org/w/index.php?title=User:Dragons_flight/math2&oldid=543253689&forceprofile=true [17:27:43] 98.14% 15.001010 1 - PoolCounter_Client::acquireForMe [17:27:51] I guess another person is looking at that [17:29:29] !log reedy synchronized wmf-config/InitialiseSettings.php [17:29:35] Logged the message, Master [17:33:39] New review: Faidon; "(11 comments)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/49710 [17:34:05] New patchset: Reedy; "(bug 45952) Reenable TimedText namespace on foundationwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53174 [17:34:14] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53174 [17:35:44] New patchset: Reedy; "(bug 45636) Namespace configuration for it.wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52449 [17:35:53] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52449 [17:38:48] heya, RobH [17:38:49] you there? [17:39:31] I see that you changed dandreescu's account in admins to ensure => absent [17:39:32] any idea why? [17:40:29] https://gerrit.wikimedia.org/r/#/c/51032/ [17:41:03] i'm going to undo that [17:41:07] lemme know if there was a reason [17:41:28] there was [17:41:42] Dan exists twice in those manifests [17:41:58] and besides that being wrong in concept, puppet also fails because we name accounts by their real name [17:42:00] but his other account doesn' exist [17:42:04] so it was a resource defined twice [17:42:16] he remvoed both accounts? [17:42:42] he deleted milimetric configs altogheter [17:42:44] New patchset: Reedy; "(bug 43524) Restrict upload on itwikivoyage to sysops" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53183 [17:42:47] but then changed dandreescu to ensure => absent [17:42:52] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53183 [17:44:04] New patchset: Ottomata; "Fixing Dan's account" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53184 [17:44:33] New patchset: Reedy; "Remove disabled ReaderFeedback" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47532 [17:44:40] dunno, I haven't seen the manifests, just recollecting from the discussion we had two weeks ago [17:44:58] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53184 [17:45:05] mk [17:45:11] just making sure it wasn't intentional [17:45:12] !log reedy synchronized wmf-config/InitialiseSettings.php [17:45:17] Logged the message, Master [17:45:44] * YuviPanda looks at /topic [17:45:57] * YuviPanda pokes Jeff_Green with https://rt.wikimedia.org/Ticket/Display.html?id=4687 [17:47:42] New review: Mark Bergsma; "(6 comments)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [17:47:44] YuviPanda: ah we need to update /topic. also I think your access request can't be done until tomorrow due to the 3-day wait period [17:48:36] Jeff_Green: oh [17:48:41] it is a 3 business days wait period? [17:48:44] yeah [17:48:59] damn [17:49:01] okay :) [17:49:10] i know, access =~ firearms! [17:49:15] :P [17:49:33] i'll fix motd as soon as I figure out who is next up [17:49:45] that's okay. Just hope it was documented (or if it is, I had read it :P) just so that I don't get heart broken every day :P [17:50:21] i think robh added a reply to that RT queue that explains things, but it went in at the end of the week [17:50:40] https://wikitech.wikimedia.org/view/Server_admin_log | Channel is [17:50:45] yeah, that's neat :) [17:51:46] thx notpeter [17:51:54] i was going to cry if it's week 3 for me [17:53:36] Jeff_Green: I did, and it did [17:53:42] and i just added to it [17:53:50] "If your access request is a shell access request, and you do not yet have shell access on the Wikimedia Cluster, please also attach your public SSH key to this request. " [17:54:11] Robh that's great--it'll help with the confusion [17:54:40] sbernardin1: you can do a dig search23.mgmt.pmtpa.wmnet @ns0.wikimedia.org [17:55:13] New patchset: RobH; "add mholmquist to jenkins admins / sudo to restart jenkins/psql on gallium (RT-4676)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52671 [17:55:28] Jeff_Green: CRY!!! CRY!!!!!!! [17:55:30] er [17:55:31] I mean [17:55:35] glad you're not going to cry [17:55:56] notpeter: i can't heear you [17:58:53] is it just me and my twisted perception of timezones or do we have a techops meeting in 3 minutes? [18:01:22] I think we do [18:01:30] ok i'm in [18:02:23] New patchset: Demon; "Updating gerrit to 2.5.2-1636-g353e384" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/52935 [18:02:46] New review: Demon; "Updated war to https://integration.mediawiki.org/nightly/gerrit/wmf/gerrit-2.5.2-1636-g353e384.war" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/52935 [19:04:50] !log patching Bugzilla for security - RT-4553 [19:04:56] Logged the message, Master [19:22:18] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf11 [19:22:23] Logged the message, Master [19:30:18] New patchset: Reedy; "enwiki to 1.21wmf11" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53194 [19:30:42] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53194 [19:45:47] Change merged: Ryan Lane; [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/52935 [19:52:45] notpeter: around? [19:52:52] New patchset: pugmajere; "Make ircecho more resilient to server disconnections." [operations/software] (master) - https://gerrit.wikimedia.org/r/53197 [19:58:02] Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/53197/ [19:58:26] hooray [19:58:37] ummm [19:58:54] why's the entire file rewritten? [19:59:13] it's only in a package, it wasn't on its own anywhere i could find [19:59:33] yeah, the source is with the package ;) [19:59:35] i could have missed it [19:59:36] hehe [19:59:53] hmmm [19:59:55] is mailing list server down? [19:59:57] is it not in gerrit? [20:00:07] weird [20:00:15] is it still in svn or something? [20:00:24] I guess it's possible I haven't updated it since then [20:01:35] hm. no. version 3.0…. [20:01:57] do i bug ops about this? confirming my email subscription times out when i click on the link and click "subscribe to list" [20:02:17] i get "no data received" [20:02:41] yurik: server itself is up - what's the exact link you are clicking [20:02:57] LeslieCarr, the one in the confirmation email [20:03:06] the page shows up fine [20:03:13] please copy that link [20:04:25] https://lists.wikimedia.org/mailman/confirm/pywikipedia-announce/08478b3574dfba320f3e86da388132ddcf6ecff1 [20:04:45] so you hit subscribe and then it times out ? [20:04:49] yep [20:05:19] btw, i suspect that it still goes through eventually - because some other conf links no longer work [20:05:27] overload? [20:06:48] hrm, can you click on subscribe right now ? i am monitoring logs [20:07:19] clicked - and it went right through [20:07:28] hah [20:07:35] of course it did because i'mw atching ;) [20:07:46] exactly :) [20:07:53] sec, let me see if there are other pending ones [20:09:08] LeslieCarr, found one, reading? [20:09:15] *ready? [20:09:23] wikitech-l [20:09:47] ok [20:10:03] what was the huge long string in the confirm link you clicked ? [20:10:45] oh mailman [20:10:49] you are so unmaintained [20:10:55] really? [20:10:58] Ryan_Lane: find out the secret home of the ircecho bot ? [20:11:03] MM3 is out (almost?) [20:11:14] LeslieCarr: no, it's a mystery :( [20:11:35] LeslieCarr, https://lists.wikimedia.org/mailman/confirm/pywikipedia-announce/08478b3574dfba320f3e86da388132ddcf6ecff1 [20:11:45] i just clicked the link [20:12:05] i meant in wikitech-l [20:12:16] i got a "shunted" message - wondering if that was the same thing you just clicked [20:12:18] LeslieCarr: http://mail.python.org/pipermail/mailman-announce/2012-December/000177.html [20:12:31] wow [20:12:41] it's been updated in the last 5 years!!! yay :) [20:13:00] yurik: basically does it end in bcbdbefebcb85 ? [20:14:24] LeslieCarr, let me try to subscribe to another list, just to have it clear [20:14:41] cool [20:16:16] LeslieCarr, just subscribed to wikidata, about to click https://lists.wikimedia.org/mailman/confirm/wikidata-l/aa8317d1eb895116e74093ff5672c6aca5449001 [20:16:22] cool [20:16:26] i'll watch for any errors [20:16:32] (grr i hope there are no errors) [20:16:34] bleh [20:16:40] no errors :( [20:16:42] haha [20:16:49] worked, although it did pause for a few seconds [20:16:58] ok, lets say it was a overload glitch [20:16:59] thx :) [20:17:03] np [20:17:08] reedy@ubuntu64-web-esxi:/var/www/wiki/mediawiki/extensions/OpenStackManager$ git pull [20:17:08] ssh: connect to host gerrit.wikimedia.org port 29418: Connection refused [20:17:08] fatal: The remote end hung up unexpectedly [20:17:08] Rage. [20:17:16] <^demon> Scheduled update. [20:17:19] Reedy: I think ^demon is upgrading [20:17:38] I was just about to comment on that [20:18:37] <^demon> It's back up now. [20:18:47] !log deploying change 52950 to OpenStackManager on wikitech [20:18:53] Logged the message, Master [20:26:59] * AaronSchulz thanks notpeter for the drink [20:28:33] YuviPanda: hey [20:28:33] what's up? [20:28:37] hey notpeter [20:28:39] I responded on RT [20:28:47] cool! [20:30:19] <^demon> !log Upgraded gerrit to 2.5.2-1636-g353e384, all's well [20:30:25] Logged the message, Master [20:31:38] ^demon: would you happen to know anything about deploying an extension off a deploy branch? (ie: CentralNotice currently deploys off of master, but I'd like to move that to deploy off of something else) [20:32:00] <^demon> Update make-wmf-branch [20:32:05] <^demon> It's in mediawiki/tools/release [20:32:16] coool [20:32:18] thanks :) [20:32:24] <^demon> yw [20:34:13] LeslieCarr: any chance Ryan can push his changes into operations/debs/ircecho? [20:34:20] I created a repo for it [20:34:30] okay [20:34:45] <^demon> Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/53173/ is trivial when you've got a sec. [20:34:53] LeslieCarr: thanks [20:34:55] someone want to sync the interwiki map? https://meta.wikimedia.org/w/index.php?title=Interwiki_map&diff=5314701&oldid=5288810 https://bugzilla.wikimedia.org/45108 [20:35:06] I can push it in, but I don't want to screw up the change authorship [20:35:25] yeah just im'ed him saying that [20:35:34] cool [20:37:01] ^demon: the weekly find/rm doesn't take care of this? [20:37:33] <^demon> That clears all logs over a week old. This one clears all sshd_log archives. [20:37:34] and why wouldn't we want to see that log? [20:37:41] <^demon> I don't want to retain those any longer than 24h. [20:37:51] ok [20:37:57] that cron is going to throw spam [20:38:00] use -f [20:38:08] <^demon> k, will amend [20:38:17] New review: Ryan Lane; "(1 comment)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/53173 [20:38:28] mentioned in comment :) [20:39:49] New patchset: Demon; "Don't keep any archived gerrit sshd logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53173 [20:40:06] New review: Demon; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53173 [20:43:03] binasher: https://gerrit.wikimedia.org/r/#/c/35139/ was updated [20:43:31] AaronSchulz: i +1'd it [20:44:51] ^demon: !!! [20:45:01] <^demon> Hm? [20:45:24] you resubmitted based on broken commit instead of fixed commit? :) [20:45:32] <^demon> fuuu [20:45:59] New patchset: Demon; "Don't keep any archived gerrit sshd logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53173 [20:46:06] <^demon> pressing save is hard. [20:46:46] better :-) [20:47:53] notpeter: are you done with the ex1 shard in pmtpa? [20:48:52] ^demon: do you want to sync the interwiki map for me? :) [20:48:59] binasher: yeah [20:49:05] <^demon> jeremyb_: How do I do this? [20:49:09] binasher: still needs heartbeat db, though [20:49:18] ^demon: have to rebuild it first... [20:49:21] let me find out [20:49:53] <^demon> Got it [20:49:53] <^demon> https://wikitech.wikimedia.org/wiki/Interwiki_map [20:49:58] notpeter: cool.. can you setup heartbeat for both sides? [20:50:04] yeah, was going to link there [20:50:31] binasher: sure [20:51:52] notpeter: i don't think this process is documented [20:52:06] jenkins-bot is sloooow [20:52:09] <^demon> Argh. [20:52:19] it probably should be! as well as some information about what it provides and how it can be useful [20:52:30] <^demon> -rw-r--r-- 1 reedy wikidev 818337 Feb 28 18:56 php-1.21wmf11/cache/interwiki.cdb [20:52:33] <^demon> Reedy: fix plz. [20:52:45] lol [20:53:00] I think I broke it so the update interwiki cache script would run [20:53:07] notpeter: would you be up for starting a wikitech page with that info as you're doing it, and i can help along the way? [20:53:13] <^demon> demon@fenari /home/wikipedia/common (master)$ ls -l php-1.21wmf1?/cache/interwiki.cdb [20:53:13] <^demon> -rw-r--r-- 1 apache apache 819091 Mar 11 20:51 php-1.21wmf10/cache/interwiki.cdb [20:53:13] <^demon> -rw-r--r-- 1 reedy wikidev 818337 Feb 28 18:56 php-1.21wmf11/cache/interwiki.cdb [20:53:19] how on earth is it 5pm [20:53:20] New patchset: Dzahn; "redirect wikiquote.net to wikiquote.org (RT-4695)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/53264 [20:53:33] IIRC I 777'd it for a couple of minutes [20:54:24] -rw-rw-r-- 1 reedy wikidev 2320694 Jan 29 04:53 trusted-xff.cdb [20:54:38] -rw-rw-r-- 1 reedy wikidev 818337 Feb 28 18:56 interwiki.cdb [20:54:38] even [20:55:02] binasher: sure! [20:55:23] bbiab [20:56:00] <^demon> Reedy: Can you run mwscript extensions/WikimediaMaintenance/dumpInterwiki.php -o /home/wikipedia/common/php-1.21wmf11/cache/interwiki.cdb --protocolrelative then? [20:56:03] New review: Dzahn; "apache-fast-test wikiquote.net.url mw1044" [operations/apache-config] (master) C: 2; - https://gerrit.wikimedia.org/r/53264 [20:56:15] <^demon> I already did it for wmf10, and I can sync both out when you're done. [20:57:11] I created /home/reedy/updateinterwikicache for this [20:57:19] Which is in a changeset in puppet I think [20:58:58] !log reedy synchronized php-1.21wmf11/cache/interwiki.cdb 'Updating 1.21wmf11 interwiki cache' [20:59:03] Logged the message, Master [20:59:23] !log reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache' [20:59:28] Logged the message, Master [21:00:06] Writing to the cache dir is slightly painful [21:03:41] New patchset: Asher; "pmtpa side of extension1 shard" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53269 [21:05:05] Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/53269 [21:07:29] !log deployed change 53267 to OpenStackManager on wikitech [21:07:35] Logged the message, Master [21:09:03] no jenkins on apache repo... [21:09:27] nevermind, works [21:10:07] Reedy: ^demon: woot, danke. had to purge the page but that worked [21:10:47] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/53264 [21:18:18] dzahn is doing a graceful restart of all apaches [21:18:58] !log dzahn gracefulled all apaches [21:19:04] Logged the message, Master [21:20:02] !log gracefulling eqiad Apaches - wikiquote.net now redirects to wikiquote.org [21:20:08] Logged the message, Master [21:35:30] New patchset: pugmajere; "Make ircecho more resilient to server disconnections." [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53276 [21:38:03] awww [21:38:10] * LeslieCarr swoons [21:39:25] New patchset: pugmajere; "Make ircecho more resilient to server disconnections." [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53276 [21:41:15] Change abandoned: pugmajere; "We put this into a different repository." [operations/software] (master) - https://gerrit.wikimedia.org/r/53197 [21:41:53] Ryan_Lane: ^^ :) [21:42:16] I would approve it but then there would be some impropriety in the code review system [21:42:21] \o/ [21:42:26] heh [21:43:00] LeslieCarr: tell Ryan thanks for the change! [21:43:08] I'll need to see if I can work this into adminbot too [21:43:17] though I'd imagine it would be a different issue there [21:43:33] :) [21:43:35] could be the same one, though, and that would rock [21:45:36] Change merged: Ryan Lane; [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53276 [21:46:31] New patchset: pugmajere; "Add the standard git-setup for wikimedia. (With appropriate tweaks.)" [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53277 [21:47:43] hey [21:47:51] anyone know if anything weird is upwith ns2? [21:47:58] I pushed out new zone files [21:48:00] it isn't responding [21:48:05] tried to restart pdns proc [21:48:07] still not responding [21:48:12] Ryan_Lane: RobH mutante [21:48:14] ^^ [21:48:20] restart till it works! :D [21:48:42] there's a small chance it locked up again when it restarted [21:48:44] i don't know , last time i touched DNS it still worked [21:48:45] it's rare, but happens [21:48:58] Ryan_Lane: I have thrice restarted it now [21:49:00] and no dice [21:49:04] lame [21:49:42] oh, there it go [21:49:47] problem solved [21:50:05] heh [21:50:07] as the old phrase goes "if something's not working, don't force it, just hit it with a bigger hammer." [21:50:12] doesn't answer though.. hmm [21:50:18] it does [21:50:27] I just searched it [21:50:56] ok. now to deploy new ircecho [21:50:56] dig wikipedia.org @ns2.wikimedia.org [21:50:59] timed out [21:51:24] mutante: wrong order ;) [21:52:03] nope, it works from localhost just not from sockpuppet [21:52:33] ok, it's fine [21:55:17] Change merged: Lcarr; [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53277 [21:59:54] notpeter: not i [22:00:06] when i pushed dns the other day was fine [22:02:17] New patchset: Ryan Lane; "Update to precise" [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53280 [22:05:52] Change merged: Ryan Lane; [operations/debs/ircecho] (master) - https://gerrit.wikimedia.org/r/53280 [22:26:14] binasher: ok, x1 has all the ptheartbeat shits now [22:26:30] so should be totally gtg now [22:26:43] woo woo [22:28:47] LeslieCarr: ircecho in repo is newest version [22:29:10] yay [22:29:12] :) [22:29:19] Ryan's kick ass [22:29:30] :D [22:29:36] so i should be able to update neon and then party with new ircecho ? [22:29:45] yep [22:29:50] I'm going to update manganese too [22:29:58] !log upgrading ircecho on manganese [22:30:05] Logged the message, Master [22:30:30] gerrit-wm: welcome back [22:30:49] !log upgrading ircecho on neon [22:30:55] Logged the message, Mistress of the network gear. [22:31:45] :) [22:36:25] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 184 seconds [22:43:25] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [22:43:55] binasher: I saw that you added db36/38 into the external loads section in db-pmtpa.php [22:44:00] anything else need to be done? [22:44:03] or can this be closed? https://rt.wikimedia.org/Ticket/Display.html?id=4111 [22:44:09] close [22:44:14] woo! [22:49:16] notpeter: http://blog.mariadb.org/mariadb-galera-cluster-5-5-29-stable-ga-released/ [22:49:44] it says "stable" :) [22:50:40] binasher: that's bullshit let's run trunk [22:54:42] New patchset: Ryan Lane; "Catch disconnected irc server exception" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/53287 [22:55:25] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Mon Mar 11 22:55:20 UTC 2013 [22:56:25] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [22:57:58] I would like to upgrade ICU today [22:59:05] I would have liked to be online at the same time as paravoid but that has not been practical over the last few days, and it really needs to be done [23:01:39] it really needed to be done a week ago, before Reedy deployed a whole lot of category collation changes [23:01:44] but better late than never [23:02:16] 3 is a whole lot? [23:03:35] Change merged: Ryan Lane; [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/53287 [23:03:48] was it only 3? I thought it used to be only ptwiki [23:04:17] ukwiki and plwiki were done over the weekend [23:04:22] svwiki is running now [23:04:26] anyway, it's no big deal, it just means the script needs to be run on more wikis [23:04:35] alright [23:04:44] hmmm [23:04:54] actually that collation script is still rubbish and broken isn't it? [23:05:10] people have been complaining about categories having broken sort order, right? [23:06:07] should have brought this up in the core team meeting I guess [23:06:38] * AaronSchulz likes it when Tim calls stuff "rubbish" [23:07:26] you know, I used to rename the trash icon on my desktop to "rubbish bin" [23:07:44] for some reason, that isn't in the localisation [23:08:56] !log upgrading adminbot on wikitech-static [23:09:01] Logged the message, Master [23:10:00] !log adminbot upgraded to 1.6.3 [23:10:05] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [23:10:05] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [23:10:05] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [23:10:05] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [23:10:05] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [23:10:06] Logged the message, Master [23:12:06] Reedy: when the ICU library is upgraded, all the wikis using a uca-* collation need to have updateCollation.php --force run on them [23:13:03] and until the script is run, newly-added category members will be missorted [23:13:20] so I would have preferred it if there was only one such wiki [23:13:23] binasher: PopulateFundraisingStatistics::updateDays is still broken, heh [23:13:33] I thought someone was fixing that? [23:13:44] i thought so too [23:13:47] pgehres: ^^ [23:14:01] i know you're not working on that stuff now, but do you remember where it was left? [23:14:31] enwiki: Authentication error connecting to 10.64.0.186 [23:14:34] binasher: fairly certain that they aren't trying to fix that bug and instead are trying to replace it entirely [23:14:36] binasher: lots of redis log spam [23:15:24] hmm, I think I see the problem [23:15:28] mwalker: ^ [23:15:45] see binashe_r's question [23:15:52] New patchset: Pyoungmeister; "Initial dump in of puppet forge mysql module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53289 [23:16:35] binasher, AaronSchulz; we're fixing it by just exporting the data to a public facing box and having limn import it [23:16:48] it seemed like less work [23:18:09] AaronSchulz: hrm, redis fails seemed to go nuts starting on march 6 [23:18:25] already have a fix [23:19:04] TimStarling: https://gerrit.wikimedia.org/r/#/c/53290/ [23:19:09] * AaronSchulz would merge that soon [23:21:06] AaronSchulz: when did that happen? [23:22:19] oh huh, at least its just spamming the logs big time but not actually failing redis access [23:22:33] i'm slightly surprised the initial change didn't break all sessions [23:23:26] Change abandoned: Pyoungmeister; "this is lame and hacky. if we're going to use something hacky and temporary, let's just use hashar's :)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52739 [23:25:56] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Mon Mar 11 23:25:46 UTC 2013 [23:26:25] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [23:27:32] f*cking gerrit [23:28:04] * AaronSchulz forgot it always chokes on conflict lines in summaries [23:29:36] hrm [23:30:06] * AaronSchulz tries again [23:31:46] New patchset: Pyoungmeister; "removing yuvipanda's user on stat1, adding on stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53293 [23:34:09] New patchset: RobH; "search23 mainboard swap" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53295 [23:35:13] binasher: probably since wmf11 [23:35:56] hrmm, shouldnt jenkins be testing my gerrit changeset? [23:36:03] * AaronSchulz wonders why gerrit takes 6 years to merge sometimes but not others [23:36:04] * RobH sees jenkins was idle, yet no testing [23:36:23] there it goes... it sat all idle for a bit. [23:36:24] odd. [23:36:44] AaronSchulz: I wonder the same. it's been slow for some things as of late [23:36:48] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53295 [23:37:16] Ryan_Lane: it's really annoying when merging to wmf branches for fixes [23:37:16] !log aaron synchronized php-1.21wmf10/includes/clientpool/RedisConnectionPool.php 'deployed b9fdedc57295a4dd1da684859871aeb734bf70fe' [23:37:23] Logged the message, Master [23:37:26] New patchset: Pyoungmeister; "removing yuvipanda's user on stat1, adding on stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53293 [23:38:08] !log aaron synchronized php-1.21wmf11/includes/clientpool/RedisConnectionPool.php 'deployed 5af8702a4b4208ac42fab2fb36da69d13f3e3b92' [23:38:15] Logged the message, Master [23:38:23] AaronSchulz: great success [23:39:04] AaronSchulz: also, looks like we'll be getting ram to upgrade rdb100[1-2] possibly as early as wednesday, they'll go to 72GB [23:39:06] that also killed some round trips [23:39:15] * AaronSchulz overheard the 72gb bit [23:39:28] * AaronSchulz is not complaining [23:41:23] binasher: I feel like there is always way too much 'Can't connect to MySQL server' spam in dberror.log [23:41:43] pgehres: re: your last question on the user_lastlogin thread, let's wait and see what tim replies [23:41:59] okay [23:42:13] AaronSchulz: i dunno what you're talking about. more connection failures, less query load! feature. [23:44:36] binasher: its sort of chicken and egg as apparently finalizing SUL is a blocker to Echo/Flow [23:44:39] yay, commons deleted container finished [23:45:02] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53293 [23:47:53] !log copyFileBackend.php finished for ceph for all originals and transcodes (plus math/captcha) [23:47:57] pgehres: that makes sense.. this entire part of mediawiki is made out of duct tape that's losing its sticky [23:48:01] Logged the message, Master [23:48:22] !log syncFileBackend.php Running for ceph for all wikis now [23:48:28] Logged the message, Master [23:52:05] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [23:52:57] ceph! [23:53:28] Reedy: xeno-ceph presence detected! [23:53:32] AaronSchulz: where's the best place to run eval.php in eqiad? [23:54:03] do we even have a bastion like fenari there? [23:54:27] we have a bastion (bast1001) but i don't think it's like fenari in that no mediawiki deploy stuff [23:54:33] right [23:55:16] bast1001 is just straightup bastion - for better or for worse [23:55:23] binasher: what do you to test? [23:56:00] I guess you can do it on a random job runner (cd into common/multiversion and use "php MWScript.php" instead of "mwscript" :) [23:56:15] why don't we have anything setup for this? [23:56:17] straight up bastion yo [23:56:25] RECOVERY - Puppet freshness on celsus is OK: puppet ran at Mon Mar 11 23:56:15 UTC 2013 [23:56:25] PROBLEM - Puppet freshness on celsus is CRITICAL: Puppet has not run in the last 10 hours [23:56:53] AaronSchulz: deploying code from within a primary datacenter.. bah! who would do that? [23:57:10] binasher: $traight $hootta [23:58:05] oh but what i wanted to test was just say, for keys named 1-100, see which memcached server they hash to with the current client [23:58:33] to then verify what tag name in twemproxy won't result in any invalidation [23:59:41] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53289 [23:59:44] but maybe tomorrow, i'm out