[00:47:06] What's the name of that service again that shows the mysql queries for production? The graphs themselves look similar to those rendered by graphite.wikimedia.org, but its not graphite.wikimedia.org (it had a layout from Twitter Bootstrap) [00:47:14] I can't find it anywhere [02:00:13] New patchset: Dereckson; "(bug 38036) Set si.wikipedia and si.wiktionary favicons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13427 [02:02:31] New review: Dereckson; "shellpolicy" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/13427 [02:58:24] Krinkle: you mean ishmael? [02:58:50] YES! [02:58:51] Thanks so much [02:58:51] Ah, or not [02:58:54] New patchset: Faidon; "ganglia: separate LVS servers into their own group" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13428 [02:58:58] https://ishmael.wikimedia.org/ [02:58:58] heh [02:59:04] yeah? why not? [02:59:10] Nope, not the one [02:59:17] well tell me what you want then ;) [02:59:23] gdash? .wm.o [02:59:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13428 [02:59:27] gash could be it [02:59:33] New patchset: Faidon; "Redirect secure.wikimedia.org URLs to proper HTTPS" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13429 [02:59:35] yes, thats the one [02:59:38] (for real) [02:59:40] Thanks :) [02:59:41] hehe [03:00:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13429 [03:00:08] i did just change the gdash config yesterday so i should be able to remember it's name [03:00:11] ;) [03:00:27] jeremyb: so what's the relation with gdash to graphite? [03:00:51] Is it aggregated? Or just an alternative display of a subset of the graphs? [03:00:56] gdash is predefined dashboards with predefined sets of graphs. and names [03:01:02] Right [03:01:03] in version control i think [03:01:11] So it doesn't do any data processing it self, then? [03:01:13] graphite is more of a free for all [03:01:26] look at the img src for your gdash images [03:01:30] right [03:02:09] also gdash is open to anyone, ishmael and graphite both require labs [03:02:23] what about the images from graphite? [03:02:57] erm? [03:03:13] note: i'm half asleep [03:03:52] well, if graphite uses Labs LDAP (throughu HTTP auth) - wouldn't users of gdash get errors? [03:04:02] (when not logged into graphite) [03:04:56] jeremyb: Do you know on which servers graphite, ishmael and gdash run? [03:05:03] I just created them on wikitech http://wikitech.wikimedia.org/view/Category:Services [03:05:16] wikitech had 0 search results for all three of those [03:05:25] (except a spurious server log msg) [03:05:34] yes, kinda [03:06:06] graphite is on fenari, that one I know [03:06:44] graphite is on fenari?! [03:06:56] it was at some point [03:07:19] well, looking [03:07:47] (took some time to fully grok...) [03:09:32] ugh, is labs broken? [03:11:22] yes, i hereby declare labs is broken [03:11:35] define labs [03:12:09] http://integration.wmflabs.org/testswarm/job/2 works fine [03:12:12] bastion1.pmtpa.wmflabs won't let me in [03:12:30] now it does... [03:12:46] anyway... [03:12:50] http://ganglia.wmflabs.org/latest/?r=day&c=bastion&h=bastion1 [03:12:53] oh well [03:14:37] ishmael's on spence (nagios), gdash and graphite are both on professor.pmtpa.wmnet [03:15:47] all 3 are proxied through fenari's apache [03:16:15] gdash and graphite then go on to varnish on professor and then to apache on professor [03:17:28] ishmael may not be puppetized. but it seems likely it has no varnish [03:17:32] err [03:17:34] nvm [03:17:40] it's spence like i said ;) [03:17:44] great [03:18:15] still it may not be puppetized, just installed on spence somehow [03:20:18] jeremyb: Is "professor" a regular production cluster server like fenari and spence? [03:20:27] afaik [03:20:36] ask racktables i guess [03:20:37] since its not on wikitech anywhere [03:20:52] jeremyb: how do I do that? [03:21:01] i assume you don't have access ;) [03:21:15] * Krinkle tries http://racktables.wikimedia.org/ - ah, that exists [03:21:22] not taking my login though [03:21:26] you could also look at it's mac address in dhcpd in version control and then see when it was added [03:21:30] right [03:21:42] there are some secrets in racktables i think [03:21:50] not that they necessarily need to be there? [03:22:19] its in ganglia, thats good enough for me [03:22:24] found it [03:22:24] Krinkle: https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=blob;f=files/graphite/apache.conf;h=ed422152c7ea1ec72e90627f80dd9d94cdf2e659;hb=c4cdd8c951c96994e474de1df9e780376b2f15b4#l17 [03:22:27] Wikimedia Grid > Miscellaneous pmtpa > professor.pmtpa.wmnet [03:22:35] that's the answer to your HTTP auth query [03:23:49] right, /render is allowed [03:24:08] did you want to mention the varnish? [03:24:25] for professor only not spence [03:24:48] I wouldn't know what I'm writing [03:24:50] feel free to edit :) [03:24:54] hehe [03:24:54] I saved the page [03:25:47] well i'm half asleep. could you ask me to edit it in ~36 hrs? ;-) is there a bot for that? [03:26:11] in #jquery-dev we have a bot listening to ?commands like ?tell [03:26:21] pretty neat [03:26:39] and pasting github links or bugzilla links, echoes the title/lastmodified of that thing [03:26:43] yeah, i've seen that [03:26:57] (not that one but same functionality [03:27:26] ) [03:27:27] gah! [03:27:31] xkcd.com/859/ [03:28:49] :D [03:28:53] nacht [03:28:55] you have that link on hot dial, don't you? [03:29:00] almost [05:59:50] New review: Tim Starling; "Looks good to me. Next time maybe consider SetEnvIf instead of "RewriteRule . -". Needs testing imme..." [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13293 [06:20:32] New review: Tim Starling; "(no comment)" [operations/debs/wikimedia-job-runner] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/11610 [06:35:14] I wish I knew why mod_status is apparently enabled and yet returns not found on our apaches (when run locally) [06:35:21] it's even set up for extended statistics [07:01:01] New patchset: Dzahn; "make fluorine use precise installer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13433 [07:01:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13433 [07:02:01] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13433 [07:02:03] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13433 [07:08:52] !log upgrading apt packages on brewster [07:08:58] Logged the message, Master [07:51:22] New patchset: Dzahn; "taking it back, fluorine supposed to be lucid" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13434 [07:51:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13434 [07:52:08] New patchset: Hashar; "send header from CS.php only for non CLI scripts" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13435 [07:52:10] New review: Dzahn; "per Tim" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13434 [07:52:13] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13434 [07:53:07] New review: Hashar; "Reapply change Ib7ef0858" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13435 [07:53:09] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13435 [09:16:49] New patchset: Hashar; "comment closing brace in long cases switch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13438 [09:17:07] grrr [09:17:25] New patchset: Hashar; "/etc/wmflabs-instancename containing instancename" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13439 [09:17:34] any ops around for two quick harmless merges in puppet please ? ^^^^^^^^^^ [09:17:56] ping apergos mutante [09:18:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13438 [09:18:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13439 [09:31:02] !log Jenkins: deployed gitsqlhaschanged patch ( d04f779 0f069c3 integration/jenkins.git ) [09:31:08] Logged the message, Master [09:49:56] * jeremyb spies a Tim-away [09:50:46] * jeremyb wonders if my change is fully deployed? ahhh, no not at all. just approved and not merged [09:50:55] !g I2c6ab07d [09:50:55] https://gerrit.wikimedia.org/r/I2c6ab07d [09:51:06] ugh, someone changed it [09:51:10] !gerrit I2c6ab07d [09:51:10] https://gerrit.wikimedia.org/ [09:51:17] !change I2c6ab07d [09:51:18] https://gerrit.wikimedia.org/r/I2c6ab07d [10:05:04] !gerritsearch I2c6ab07d [10:05:29] or it's just a different brain [10:11:01] !change del [10:11:02] Successfully removed change [10:11:06] !g del [10:11:06] Unable to find the specified key in db [10:11:07] jeremyb: I think I did to make it use shorter url :-D [10:11:16] did not though about looking up for the I [10:11:26] !g [10:11:30] huh [10:11:41] !gerritsearch is https://gerrit.wikimedia.org/r/#q,$1,n,z [10:11:42] Key was added [10:11:52] so hmm I2c6ab07d is in gerrit [10:11:54] but not merged [10:12:03] pending ops review + merge + deploymenet [10:12:10] I am not sure how they handle apaches changes [10:12:10] ops review? [10:12:19] operations team [10:12:25] they are responsible for the apache confs [10:12:26] i think it just needs someone standing by to test and revert [10:12:30] i can test some [10:12:57] !g alias gerritsearch [10:12:57] Alias is already existing [10:13:08] !g unalias [10:13:08] Alias removed! [10:13:10] !g alias gerritsearch [10:13:11] Created new alias for this key [10:13:14] !change alias gerritsearch [10:13:15] Created new alias for this key [10:13:32] so many keys... [10:13:48] !gerritchange alias gerritsearch [10:13:48] Created new alias for this key [10:14:04] !gitchange alias gerritsearch [10:14:04] Created new alias for this key [10:14:08] !githash alias gerritsearch [10:14:08] Created new alias for this key [10:14:22] !hash alias gerritsearch [10:14:23] Created new alias for this key [10:14:26] !sha1 alias gerritsearch [10:14:27] Created new alias for this key [10:14:36] !hash I2c6ab07d [10:14:36] https://gerrit.wikimedia.org/r/#q,I2c6ab07d,n,z [10:14:41] ok, done [10:15:04] hashar: are you sure it was you? i think it's really just a different brain [10:15:15] now i synced the brains for that key [10:15:23] \O/ [10:15:27] (well these 2. not all brains) [10:16:08] hashar: do those RTs listed look relevant? [10:17:32] jeremyb: looks like (I have read the three of them) [10:17:44] good ;) [10:18:51] New review: Dzahn; "see this one https://bugzilla.wikimedia.org/show_bug.cgi?id=36352 . that asks for PROJECTNAME as an ..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/13439 [10:19:33] mutante: you can use /bug \d+/ [10:19:49] in gerrit comments? ah, nice [10:19:55] surely [10:20:04] you can even help fix them if you like ;) [10:20:14] heh, i created it [10:20:21] !g 11589 | mutante [10:20:22] mutante: https://gerrit.wikimedia.org/r/#q,11589,n,z [10:20:22] but sure [10:20:54] mutante: that PROJECTNAME is a different change I guess [10:21:25] hashar: but it mentions how we already have $INSTANCENAME. found that out when wanting something like you intended now [10:21:43] hashar: i wish gerrit didn't 404 quite as much as it does (i.e. if the short version just DTRT) and used pushstate [10:22:50] mutante: I will add a patch to bashrc [10:23:22] !g 6118 | hashar [10:23:22] hashar: https://gerrit.wikimedia.org/r/#q,6118,n,z [10:24:04] mutante: I am pretty sure we can abandon 6118 [10:24:26] we have misc-scripts installed on 'beta' [10:24:40] hashar: thanks @ bashrc, and yea, well, i told petan its ok to wait for the bug :) [10:24:41] so logging is done with /usr/local/bin/log [10:26:12] mutante: definitely abandon 6118. I have added a comment on it [10:26:20] hashar: could you put that comment on 6118, i think nobody sees it but me all the time, because i have been added as reviewer and even in the commit message ;) [10:26:33] thanks! [10:26:50] haha "Added a script for mutante to fix" [10:27:30] heh,yea, and sitting in "My" link of course [10:29:10] mutante: I did comment on 6118 :) [10:29:14] New patchset: Hashar; "(bug 36352) publish instanceproject as an env variable" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13444 [10:29:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13444 [10:31:09] mutante: are you in the mood for 13293 ? [10:31:26] * jeremyb will be around another ~20-30 mins for testing [10:31:49] not on a friday :) [10:31:52] I would say [10:31:56] haha [10:32:57] * jeremyb abstains from friday opinion in this instance [10:33:22] New review: Dzahn; "looks harmless and could possibly keep somebody from changing stuff in wrong realm." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13438 [10:33:24] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13438 [10:36:42] jeremyb: ehm, i was looking at the gerrit commentlink patterns, "very lightly tested"?;) [10:36:57] i mean, it won't break existing links i guess [10:36:58] mutante: gerrit.pmtpa.wmflabs [10:37:06] mutante: it's -1'd for a reason [10:37:10] ;) [10:37:18] heh,ok, wont touch:) [10:37:37] but you can make further changes if you like ;) [10:38:10] sure. but i need to find "missing" harddisks [10:38:12] though i was surprised to find that changes are retroactive. so it's not so bad if you break links because they can be fixed [10:38:13] heh [10:38:36] errrrr, as in not in /dev/ but installed? [10:39:23] mutante: I also got /etc/wmflabs-instancename to publish openstack instancename https://gerrit.wikimedia.org/r/#/c/13439/ [10:39:27] as in "it's not there per fdisk/parted/dmesg, but in BIOS there is an "Unknown device" and after next reboot you have an sdb again [10:39:57] and publish instance project in env (your bug 36352) is https://gerrit.wikimedia.org/r/#/c/13444/ [10:41:45] mutante: and then? disappears again? [10:41:48] ok, i will look for these, but the "redirects.conf" i would indeed prefer not right now then :) [10:42:31] jeremyb: and then i am trying UEFI vs. BIOS boot mode just to see and thats where i was [10:43:25] mutante: what about echo "- - -" > /sys/...../scsi/rescan ? [10:43:44] mutante: was it plugged in while up? [10:44:37] i think it's usually in /sys/class [10:46:56] New review: Jeremyb; "consensus seems to be not on a Friday. please come poke me when you're ready so I can test it." [operations/apache-config] (master) C: 0; - https://gerrit.wikimedia.org/r/13293 [10:48:58] New review: Dzahn; "thanks. i requested this in /bug 36352" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13444 [10:49:01] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13444 [10:50:07] mutante: no / required [10:50:36] /foo bar/ is just a way to indicate it's a regex [10:50:57] eh yeah, that was more a typo than intended [10:51:24] removing the URL i had there again first and then stopped myself:) [10:52:45] hashar: right, reading that commit message again you already pointed out there is the variable [10:53:32] yup i checked the openstack extension [10:53:40] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13439 [10:53:41] it does provides several instance variable [10:53:42] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13439 [10:53:49] might end up writing some json/xml files or something [10:54:02] and put that in /etc/wmflabs/instance.json [10:54:53] good thanks mutante! thus I will be able to have a nice server name in beta mediawiki (aka deployment-apache30 instead of i-0002A) [10:54:56] ok for now? than i'll go back to where i was [10:55:14] good luck with scsi :) [10:55:21] kk, thx, cya [10:55:28] and thanks for the reviews! [10:55:55] thanks for solving the bug (i did not test on labs instance yet though) [10:56:11] feel free to close it [10:56:22] +export INSTANCEPROJECT=deployment-prep [10:56:24] looks like it works [10:56:48] jeremyb: i'll try the rescan. i don't know how it was actually put in there, i just see in a ticket they should be installed. bbl [10:56:48] $ echo $INSTANCEPROJECT [10:56:49] deployment-prep [10:57:22] :) [11:01:16] petr's sending mail but not on IRC? [11:01:23] * jeremyb runs away [11:08:57] runtoo [11:08:59] lunch time [12:22:20] back [12:23:47] Did someone just set up gerrit notifications to mediawiki-commits ? [12:24:11] I just got the first one I think [12:25:04] I feel a bit silly [12:25:07] but [12:25:24] oh? [12:25:28] :O [12:25:32] @replag [12:25:33] Krinkle-away: No replag currently. See also "replag all". [12:25:36] @externals [12:25:36] Krinkle-away: [operations/mediawiki-config.git] Checked out HEAD: d9439d6473d5469d0e27e5c11248947af1e18007 - https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=commit;h=d9439d6473d5469d0e27e5c11248947af1e18007 [12:25:37] @externals update [12:25:59] but when I go to wikisource.org [12:26:12] English, French etc. link to en.wiki*pedia*.org [12:26:15] Krinkle-away: Successfully updated externals! [12:26:19] @externals [12:26:19] Krinkle-away: [operations/mediawiki-config.git] Checked out HEAD: 377ba926b047b54262b92eb3253e2e4610e67691 - https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=commit;h=377ba926b047b54262b92eb3253e2e4610e67691 [12:28:07] Krinkle-away: yes, demon just found out and is the new list admin [12:28:36] hashar: ^ [12:28:43] o [12:29:14] paravoid: do we get an issue with interwiki links? [12:30:09] I am not sure what you expect from me [12:30:42] confirmed. wikisource.org links to wikipedia [12:30:55] http://wikisource.org/wiki/Main_Page [12:31:10] then go to a language version [12:31:54] template mess-up? [12:32:03] hashar: your experience :-) [12:32:19] should be http://wikisource.org/wiki/Template:TopTenCircle [12:32:35] not edited since april [12:32:48] fucking templates [12:32:49] seriously [12:32:54] :-) [12:33:02] that then refers to the interwiki links [12:33:03] grr [12:33:04] dont have edit rights:) [12:33:13] (sorry for swearing) [12:33:35] thanksfully [12:33:50] someone solved a multiple years bug about displaying interwiki links [12:33:54] so we can just look at : http://wikisource.org/wiki/Special:Interwiki [12:34:00] aa points to aa.wikipedia.org [12:34:07] so someone messed up the interwiki table [12:34:12] good news : it is easy to fix [12:34:20] bad news : I have no idea how to update interwiki tables on the cluster :-( [12:35:02] hmm, as long as its the right db ... looking [12:35:16] <^demon> Should be dumpInterwiki.php, then sync out the new cache file. [12:35:18] did it manually on wikitech before [12:35:20] <^demon> We don't use the interwiki table. [12:35:24] ooh [12:35:25] there is https://wikitech.wikimedia.org/view/Update_the_interwiki_cache [12:35:32] reference /home/wikipedia/common/php/cache/interwiki.cdb [12:38:03] !log dumping interwiki and updating interwiki cache (to fix broken interwiki links, like wikisource.org -> wikipedia.org) [12:38:09] Logged the message, Master [12:38:25] mark: want to review https://gerrit.wikimedia.org/r/#/c/13429/ ? [12:38:30] yea, i think did that before [12:39:10] * hashar looks at http://meta.wikimedia.org/wiki/Interwiki_map [12:39:35] how do you dump the contents of the .cdb file to stdout [12:39:49] I don't understand what's wrong tbh [12:40:03] but I'm fine not knowing too [12:40:08] if it's easier for you to fix it than explain to me :) [12:40:36] mutante: have you updated the cache? If so just sync the interwiki.cdb file [12:40:37] paravoid: what about serving secure.wikimedia.org on the https infra instead? [12:40:49] hashar: i did both [12:41:30] mark: that's the next step, but it's friday :) [12:41:59] also I'll be away the next week and the week after that [12:42:13] and didn't want to make a mess then leave [12:43:05] hehe [12:44:16] soo wel [12:44:23] I have no idea what might be wrong :( [12:44:29] hashar: they use just [[:en::Main Page [12:44:41] hashar: so which prefix is that supposed to be? "self"? [12:44:56] so on a wikisource [[:en:FOO]] should expends to en.wikisource.org/wiki/FOO [12:45:07] it seems the expanding mechanism is broken [12:45:12] but on wikipedia the same link expends to wikipedia.org domain [12:45:18] not the actual interwiki table [12:45:41] or does it have an entry for the self-reference [12:46:08] mark: serve secure, or send redirects? [12:46:12] I'd much prefer the latter. [12:46:42] Ryan_Lane: oh you're here [12:46:42] New review: Helder.wiki; "(no comment)" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/12185 [12:46:49] New review: Mark Bergsma; "Think that should work, but test carefully." [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/13429 [12:46:50] yes, and I feel like crap :( [12:47:02] * Damianz makes Ryan_Lane a cookie [12:47:03] Ryan_Lane: serve redirects on the new https cluster? [12:47:32] move secure.wm.o there, but it should only have rewrite rules to redirect people to where they shoul dbe [12:47:35] imo, step 1) serve redirects from singer, step 2) move secure *and* redirects to nginx [12:47:41] no [12:47:49] why write the rewrite rules twice? [12:47:55] singer is apache [12:48:00] I wrote that already [12:48:01] for apache [12:48:05] see above [12:48:07] hashar: just put an "s:" there in Template:TopTenCircle for now? [12:48:08] you sure you did it correct? [12:48:16] of course not [12:48:23] it's not a simple 1:1 mapping [12:48:27] that's why I'm asking for reviews [12:48:32] Ryan_Lane: https://gerrit.wikimedia.org/r/13429 [12:48:35] i'm really not gonna check your mod_rewrite stuff ;) [12:48:41] I really don't see the point [12:49:02] the regexps and nginx vs. apache are totally orthogonal [12:49:06] heh [12:49:10] this is very unlikely to work [12:49:14] mutante: hop should not be needed. On a wikisource wiki, [[:en:]] should points to en.wikisource.org [12:49:16] how come? [12:49:17] paravoid: ask Roan [12:49:21] !g 13293 | Ryan_Lane [12:49:21] Ryan_Lane: https://gerrit.wikimedia.org/r/#q,13293,n,z [12:49:26] okay, will do [12:49:32] <--i have been asked earlier about that [12:49:33] because we have really stupid things set up [12:49:52] btw, I looked at https everywhere's config before we switched [12:49:58] and it's just that [12:50:13] also, I've run those regexps on singer's access log [12:50:20] excluded those and then looked at the rest [12:50:28] yeah, but we have strange url structureres [12:50:33] Ryan_Lane / paravoid: that was just for the redirect/https [12:50:35] *structures [12:50:44] i can't say I would mind much if secure redirects would break a bit [12:50:47] mutante: switching to -dev [12:50:50] agreed [12:50:50] all the more reason for people to switch over :P [12:51:24] Ryan_Lane: discuss it later with Roan? [12:51:27] New review: Ryan Lane; "(no comment)" [operations/apache-config] (master); V: 0 C: 1; - https://gerrit.wikimedia.org/r/13293 [12:51:30] paravoid: yeah [12:51:41] Ryan_Lane: in any case, the plan is to move it to nginx as the next step [12:51:46] he's been wanting this for a while, and knows how to go about it [12:51:56] it'll be way less likely to break if you do it on nginx first [12:52:01] why? [12:52:09] since you can test it before you move secure.wm.o over? [12:52:15] I can test it now too? [12:52:31] also, nginxs don't have access.log [12:52:40] singer does? [12:52:43]