[00:01:23] I'm a n00b, so I don't know if icinga is something active, or what, but it's reporting 502 Bad Gateway at http://icinga.wmflabs.org/ [00:02:35] danorton: i think Damianz is on it , in #wikimedia-labs [00:02:39] because that's labs monitoring [00:02:56] k. Just browsing and figuring stuff out, today. [00:02:58] and he just mentioned it has an issue, though unrelated to us just merging a monitoring change as well [00:03:16] Yeah - I know about that issue, small problem logging in to fix it right now. Will get to it shortly [00:04:28] danorton: thanks for reporting though, that is the monitoring for labs.wikimedia.org instances [00:04:55] Question: where do I find enwiki server config info, esp. php.ini and loaded modules (e.g. gmp)? [00:06:47] danorton: in the mediawiki-config repo [00:06:55] or puppet [00:07:15] or wikimedia-task-appserver [00:08:50] Most of that should be in puppet.. [00:09:08] git clone https://gerrit.wikimedia.org/r/operations/debs/wikimedia-task-appserver [00:09:39] cd ./debian/wikimedia-task-appserver/etc/php5 [00:09:53] because you said PHP config [00:10:17] * matanya falls asleep on the keyboard [00:10:28] thx mutante [00:10:39] hmm, i just realized there seems to be nothing really in there [00:10:42] so i might be wrong :p [00:10:58] anyways, that is the Debian package we install on all the application servers [00:11:23] mutante, I need to access caesium.equiad.wmnet. I should have the neccessary rights, but when I try to proxy through bast1001, all I get is a blank screen. Can you help me? [00:12:10] danorton: some in modules/applicationserver/files/php/ in the puppet repo [00:12:46] mglaser: i can confirm your user exists on caesium, i see how it has been created in logs [00:12:55] mglaser: what i don't see is a failed login attempt so far [00:13:06] wanna try again while i watch log now? [00:13:13] yes, sec [00:13:18] mglaser: ssh -vvv would help too [00:13:32] just tried [00:13:32] reedy logs in:) [00:13:52] mutante: https://gerrit.wikimedia.org/r/#/c/109950/ Doesn't need building/deploying, but getting it out of the repo would be good (moved to puppet and changed location to /srv/scap)... [00:14:08] mglaser: Set up ProxyCommand? [00:14:31] mglaser: yes, what hoo says, want a config example? [00:14:43] that would be great [00:14:50] Host fluorine [00:14:50] IdentityFile ~/.ssh/wmfProd.private [00:14:50] User hoo [00:14:50] ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org [00:15:27] that's very similar to what i have [00:15:40] that's not for putty, right? [00:15:43] ;) [00:15:51] oh, true [00:15:55] OpenSSh that is [00:15:58] eh, no:) but in putty you also have ProxyCommand option [00:16:01] hold on [00:16:08] i have this: plink.exe bast1001.wikimedia.org -l mglaser -agent -nc %host:%port [00:16:16] http://mikelococo.com/2008/01/multihop-ssh/ [00:16:17] as a "local proxy commant" [00:16:46] s/t/d [00:17:34] eh, yea, it looks about right but i don't have more details why it fails [00:18:02] somehow it doesnt even get to caesium though [00:18:55] looks for the wikitech page about this..brb [00:19:35] mglaser: https://wikitech.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP [00:19:46] that is for labs . to get to instances behind labs bastion [00:19:49] mh... you might miss -i [00:19:52] but that should mostly work here too [00:19:58] if you replace the bastion name etc [00:20:20] https://wikitech.wikimedia.org/w/images/5/5b/20130118-2224-PuTTY_Configuration.png [00:20:56] eh, wrong one, this one https://wikitech.wikimedia.org/w/images/5/5b/20130118-2224-PuTTY_Configuration.png [00:21:30] hmm, that's what you did :P [00:22:25] kind of [00:23:08] wait i also don't see your attempts in log on the bastion [00:23:35] this is bast1001.wikimedia.org, right? [00:23:37] can you just connect to that first, without even trying to proxy [00:23:39] yes [00:23:44] mutante: Did you force run puppet after the keys were added? :P [00:23:45] i'm watching that too now [00:24:18] hoo: the key was already on caesium, and i saw the user creation in logs [00:24:26] Ok, well :/ [00:26:22] i'm on bast1001 now [00:26:31] mglaser: typo! [00:26:32] Invalid user mglaer [00:26:32] session opened :) [00:26:39] ok, then try the proxy again with the fixed username? [00:26:50] it works:) [00:26:57] saw you log in, good [00:26:57] Yeah! [00:26:59] thanks! [00:27:02] yw [00:27:44] Reedy: ok, so you say that really doesnt need building?:) [00:27:50] then yea, can do [00:27:50] it seems bast1001 fingerprint was not stored, so it didn't work [00:28:37] ah, so yeah, that initial connect might help with that [00:29:02] it seems to be the case [00:47:16] mutante, sorry, that's a real beginners question: I need to sign the tarballs. I can do gpg --sign... but where do people get my public key from? [00:47:52] mglaser: either from a keyserver you upload it too, and/or it should be added to the keys.html file [00:48:32] mglaser: http://www.mediawiki.org/keys/keys.html [00:48:41] you should be on that when you sign mw releases [00:49:32] mglaser: before you upload to a keyserver, make sure it's not valid until eternity, and you have revocation certs made and stored somewhere [00:50:07] that keys.html file used to be on noc.wm, not it's on mediawiki.org so i'm not completely sure who adds that for you [00:50:50] I see [00:51:21] mutante: mglaser: operations/mediawiki-config repo, docroot/mediawiki/keys [00:51:42] add to keys.html and keys.txt, commit, push, review, pull, sync-docroot [00:51:52] :) [00:51:59] ah. Reedy, thanks [00:57:56] mutante, would 10 years be a reasonable validity period? [01:00:03] mglaser: feels to long personally, i'd just do 2 [01:00:05] too [01:00:30] two :p [01:01:48] :) [01:13:36] mutante, need to bug you again. I uploaded the file, but when I try to access it via http://downloads.wikimedia.org/mediawiki/1.22/, I get redirected to dumps.wikimedia.org and the tarball is not there [01:13:52] what am I doing wrong? [01:14:16] owner is mglaser. owner of all the other files is root [01:14:55] eh., can't find the server at downloads.wikimedia.org. [01:15:26] when was that added? today? [01:15:51] mglaser: i have no idea yet, wasn't involved [01:16:13] so where exactly on caesium are you [01:17:33] srv/org/wikimedia/releases/mediawiki/1.22 [01:17:49] mglaser: it used to be http://dumps.wikimedia.org/mediawiki/1.22/ [01:18:05] right [01:18:18] and I guess there's a redirect in place [01:18:49] well, the permisssions look odd [01:18:58] all the other files are owned by root:mwupld [01:19:12] want me to fix that? [01:20:02] that'll be great [01:21:18] mglaser: how about the .sig files [01:22:18] !log fixing permissions for 1.22.3 files on caesium, let mwupld group own them like all the other files [01:26:29] hm. still can't see the file [01:26:30] !log same for 1.22.3 sig files that were owned by mah. find . -uid 1232 -exec chown root:mwupld {} \; [01:26:43] mglaser: ^ i saw the sig files, fixed as well [01:26:44] yes, hexmode is siging [01:27:06] it's late and it takes too long to set up my keys ;) [01:28:03] mumbles [01:29:56] mglaser: ?? [01:30:10] hexmode [01:30:32] so, if you need bed, I can follow up with ops here [01:30:35] mglaser: what's up? [01:30:40] but if you are in the middle of it [01:30:44] ... [01:31:04] csteipp: tarball is on caesium and is signed [01:31:26] hrm... but dl.w.o doesn't point there yet [01:31:28] caesium = new download server? [01:31:29] ? [01:31:34] yep [01:31:42] oh, isn't it release.wm.o? [01:32:01] hexmode, csteipp, mutante, does it make sense to upload the other tarballs (1.21, 1.19) as well? [01:32:02] nope, guess not. [01:32:11] i dunno [01:32:22] csteipp: yep, you're right [01:32:23] mglaser: Yes, we need all the tarballs [01:32:24] there is still http://dumps.wikimedia.org/mediawiki/1.22/ [01:32:32] but i have no idea what happened with the new server [01:32:37] mglaser: use release.wm.o in the email [01:32:47] https://releases.wikimedia.org/mediawiki/1.22/ [01:33:08] ah, there you go:) [01:33:16] well, then, i just had to fix some permissions [01:33:18] hexmode: [01:33:34] mutante: it works! [01:33:46] * hexmode does his happy dance [01:33:46] cool. [01:34:03] i didn't know it was releases.wm [01:34:11] but when you upload to caesium [01:34:12] glad mglaser broke it in for me so I don't have to deal with all this next month :P [01:34:19] all the other files are owned by "mwupld" user [01:34:21] which is new [01:34:24] eh, s/user/group [01:34:25] mutante, can you change the permissions on /srv/org/wikimedia/releases/mediawiki/1.21 and /srv/org/wikimedia/releases/mediawiki/1.19 so that I can upload the files? [01:34:48] that is, rwxrwx-x [01:35:31] mglaser: upload them to your home dir for now, unless mutante is super fast :) [01:35:52] 4.0K drwxr-xr-x 2 root mwupld 4.0K Jan 28 18:50 1.21 [01:36:00] 4.0K drwxr-xr-x 2 root mwupld 4.0K Jan 28 18:51 1.19 [01:36:05] looks right [01:36:07] without changes [01:36:07] he is :) [01:36:14] unless you are not in that group [01:36:22] g+w? [01:36:23] but you are [01:36:28] mutante: g+w, yeah [01:36:30] oh, of course [01:36:50] try again [01:37:16] uploading 1.21.6, 2% [01:37:59] !log allowing group mwupld write in ./releases/1.19 and 1.21, the other dirs were already like that [01:38:09] yea, now it's consistent [01:38:10] Logged the message, Master [01:39:31] wait, not really all ..1.16 and 1.17 too. fixed [01:40:42] 32% [01:41:52] mglaser: you should have the honor of doing !log mediawiki 1.22.3 is out or something when it's done and confirmed [01:42:04] :) [01:42:05] greg-g loves that,heh [01:42:22] mutante: could you install socat on there so I can fwd gpg-agent? [01:42:35] hexmode: not manually, sry [01:42:50] mutante: apt-get? [01:42:59] hexmode: nope, just puppet [01:43:02] :`( [01:43:17] I'm gonna have to make a puppet commit, aren't I? [01:43:50] lame, I long for the days of cowboys in the server room [01:44:12] hexmode: try JeffGreen, i am already fixing something i wasnt involved in [01:44:23] mglaser: let me know when you've uploaded them [01:44:40] mutante: I can use the gerrit [01:45:08] cool [01:45:14] thanks [01:45:17] Forwarding on your signing key seems... kinda scary [01:45:35] that is another reason why it should be gerrit:) [01:45:46] add csteipp as reviewer, heh [01:47:07] We should start putting ops policies in gerrit, so I can review them :) [01:47:22] :) [01:47:27] 90% [01:47:37] csteipp: policy 1 - no manual installs of anything :) [01:47:51] +2 [01:48:02] csteipp: fine, no scariness for you! [01:48:49] uploaded [01:48:56] so only 1.19 to go [01:49:19] mglaser: signing 1.21 [01:51:49] 1.21.6 signed [01:53:20] yay! 1.19: 37% [01:53:43] I'm going to update the release notes on mw.o [01:53:53] (after a cigarette break) [02:00:24] hexmode, upload is done [02:00:35] brb [02:00:57] mglaser: signing is done :) [02:01:31] so cool! [02:03:32] mglaser: release email looks fine [02:05:28] hexmode, I'll send it [02:08:37] mglaser: congrats, cya later [02:09:09] mutante, thanks so much for your help! [02:09:16] cya [02:12:49] looks like web frontend for wikitech.wikimedia.org is down [02:13:19] csteipp, is mw-announce forwarded to other lists? [02:13:50] mglaser: no. have to do all 3 in the To [02:14:45] ok, thanks [02:16:07] csteipp, mail is out [02:17:38] stable release date would be UTC or SF time? [02:18:49] mglaser: I do Pacific, but no standard :) [02:21:47] OK, did so [02:22:33] mglaser: approved [02:24:00] csteipp, thx [02:24:33] oh, csteipp, there's no patch file for security-only :/ [02:27:00] git format-patch hash..hash? [02:28:16] I'm pulling down the repo.. I should be able to get that [02:29:08] You've got mail! [02:29:48] git format-patch f560223.. b447e34 gives me 4 patch files [02:33:00] Hello71, yes. Only 4 by now, but counting... [02:35:30] ok, it's 3:30 am, I guess my work is done for now. [02:36:05] csteipp, if you could to the hash..hash, cool. If not, drop me a note and I'll to it after sleep [02:36:52] hexmode, csteipp, mutante, Reedy, hoo, thanks so much for your help! [02:36:52] g'n8 [02:37:13] good night ;) [02:38:51] night [02:40:49] oh, wait [02:41:37] on mw.o, interwiki link download: points to http://download instead of http://releases [02:43:30] what can we do about that? [02:44:26] use an external link? [02:45:31] That might work, but I think the templates would have to be updated too.. [02:45:53] But if you can do it, it would be good to get the links working now [02:45:59] We can do it right later. [02:46:00] who can adjust the interwiki link? [02:46:03] k [02:46:10] I don't actually know that one [02:46:30] oh, I do [02:46:56] hoo, you're the man [02:47:07] I think that's the best way [02:47:16] mglaser: so [[download:foo]] shall point to? [02:47:35] http://releases.wikimedia.org/foo ? [02:47:46] http://releases.wikimedia.org/ [02:47:50] right [02:51:18] hoo, drop me a note when you're done [02:56:59] mglaser: I probably need Reedy for that, as I doubt I can deploy that myelf [02:57:01] * myself [02:57:04] as it uses a cdb [02:57:34] k [02:57:52] btw, mglaser: git format-patch 3ac51b1..1edd68b --stdout > /tmp/SecOnly.patch [02:58:17] And that applies cleanly on 1.22.2 [03:00:57] csteipp: Can you replace the interwiki.cdb [03:01:26] hoo: I'm actually not sure how to do that [03:01:27] mwscript extensions/WikimediaMaintenance/rebuildInterwiki.php can build the new one [03:02:26] * csteipp looking [03:02:58] hoo, csteipp, I updated Template:DownloadMediaWiki and Template:MW_stable_release_link for a workaround solution [03:03:16] csteipp: sudo -u mwdeploy mwscript extensions/WikimediaMaintenance/rebuildInterwiki.php --wiki metawiki -o wmf-config/interwiki.cdb [03:03:20] and then deploy that [03:03:22] I guess [03:03:38] interwiki.cdb is in version control though... I can also submit it as patch, probably [03:03:50] ok, I'm off [03:03:53] cu [03:06:52] hoo: Since markus changed the template, I think I'm going to let Reedy handle that one. Since no one else is around if things go bad.. [03:07:04] csteipp: He already did it :) [03:07:28] haha. Dang, Reedy is way too fast sometimes :) [03:08:21] Ok, now it really is time... good night [03:08:27] night! [03:29:51] Anybody know what's up w/ wikimedia/operations-puppet.git ? [03:29:57] Failed twice on clone [03:29:59] warning: remote HEAD refers to nonexistent ref, unable to checkout. [04:26:29] danorton: Still having trouble? [04:26:45] If so, can you please paste the exact command you're running? [04:26:56] Thx Gloria, problem on my end. [04:27:03] There is no HEAD on that repo [04:27:38] I think it's origin/production instead of origin/master? [04:27:59] I checked out from production [04:28:11] By default, git clone checks out from HEAD after downloading. [04:29:22] I was curious to see what PHP extensions are loaded dynamically. [04:29:33] Now I'm looking to see which are built-in. [04:29:38] Not sure where to look for that. [04:29:47] I'm specifically curious about gmp [04:30:12] You're looking for the PHP config? [04:30:41] Yes, let's thank Gloria. [04:31:00] It would be the build config, not the run config [04:31:42] The build config is blank regarding extensions (i.e. none are loaded dynamically, which improves load performance) [04:32:47] https://noc.wikimedia.org/conf/ --> operations/puppet.git, maybe. [04:33:14] Betacommand: is there a way to organize categories by byte size of page? [04:33:47] Newyorkadam: Do you have a Wikimedia Labs account? [04:33:53] You can just query this information yourself. [04:33:53] Gloria: no [04:33:58] Go make one. :-) [04:34:14] I'd rather spend the time teaching you how to use a Labs account. [04:34:27] don't i need to request one? [04:34:37] You should be able to self-register. [04:34:53] https://wikitech.wikimedia.org/wiki/Main_Page#Labs right? [04:35:41] Gloria: I need a token to register? [04:36:58] Newyorkadam: Yes to the first question. No to the second. [08:11:53] How can I measure how much time a particular extension's resources are taking to load? [09:07:29] (Search for "profiling" on mediawiki.org?) [09:11:20] Nemo_bis: thanks [11:43:12] is wikipedia site hosted at one server or mirrored across the world? [11:43:38] its rather popular to shove a large site onto multiple boxes but i dont find any detail about wikipedia being like that [11:47:58] gry: There are servers at different sites... our main DC is in Ashburn, and there are caching nodes in San Francisco and Amsterdam. Also there's a secondary data center in Tampa [11:48:57] 3 in US, 1 in EU [11:49:14] i'm in Australia, many things hang on waiting for bits.wikimedia.org [11:49:20] additional waits [11:49:30] crappy wifi [11:49:53] what does somebody means when says: "Sharing an early patch on Gerrit for review is welcome!"? how can I do this? [11:51:24] https://www.mediawiki.org/wiki/Gerrit/Tutorial#How_to_submit_a_patch [11:51:36] https://www.mediawiki.org/wiki/Gerrit/Getting_started [11:54:13] thanks.. I thought that could be a patch other than the PATCH_TO_REVIEW... [20:27:07] is this a joke [20:28:54] maybe [20:29:00] wrong chat :p [20:29:01] depends on what [20:29:10] :) [20:30:31] about lost not being on on demand or viera cast [20:55:59] MatmaRex, rdwre_ recalls that you run a bot that looks at a wiki page which is a list of repos, users, and regexen -- and then looks at the review stream adding each user to each review that matches [20:56:03] does that ring a bell at all? [20:56:26] mwalker: yes, but i'm not the one that does this [20:56:36] do you know where that page lives though? [20:56:53] i'm finding out [20:57:00] would be easier if search wasn't down :P [20:57:23] mwalker: gerrit has a feature like that natively, though [20:57:49] ah; that would explain why I wasn't getting results :p [20:58:04] mwalker: https://gerrit.wikimedia.org/r/#/settings/projects i use that myself [20:58:29] yep yep [20:59:23] mwalker: https://www.mediawiki.org/wiki/Git/Reviewers [20:59:43] why do we have everything split under "Git/" and "Gerrit/"? :/ [20:59:55] I don't know; that might also be why I couldn't find it easily [20:59:58] thanks for your foo! [21:04:56] mwalker: it's my bot, actually :-) [21:05:25] its very useful; but also very hard to find :p [21:05:39] at least with my five mintues of looking [21:05:47] I can imagine, yes. Any suggestions on improving that are welcome :-) [21:06:00] i wonder if anyone would be mad if i moved Git/* to Gerrit/*? [21:06:06] probably not [21:06:11] well, I would [21:06:18] because I'd have to adapt the bot ;-) [23:32:21] bd808: we reverted the change causing that infinite lopp: https://gerrit.wikimedia.org/r/#/c/116174/2 [23:32:33] should that go out today or is it safe to wait for lightning deploy on monday? [23:33:44] awjr: MaxSem and I were sort of talking about that in another channel. greg-g is going to make the call [23:33:52] kk [23:34:00] * bd808 got lost in the channels [23:34:21] hehehe where are you guys chatting? [23:34:22] awjr: bd808 (and no maxsem) yes, let's do it [23:34:28] mediawiki-core [23:34:33] @_@ [23:34:36] i didnt even know that was a channel. [23:34:44] it's about 4 months old or so