[01:57:48] i see mail from gerrit that says it has cc'd noc@wm.o; is that really spamming the OTRS queue? [01:58:38] mark set it up like that i believe [01:58:42] so its hardly spam [01:59:04] but maybe it's creating new tickets for each msg or maybe not [01:59:23] and maybe the gerrit prefs for that user could be changed to not send so much mail [01:59:56] i think spam is kinda accurate. even if it is actually solicited [02:00:09] Maybe he just has a fetish for spam [02:00:40] p858snake|l: in berlin? what's your home TZ? [02:01:15] lolno [02:13:39] !log LocalisationUpdate completed (1.20wmf4) at Sun Jun 3 02:13:39 UTC 2012 [02:13:46] Logged the message, Master [02:35:32] !log LocalisationUpdate completed (1.20wmf3) at Sun Jun 3 02:35:31 UTC 2012 [02:35:35] Logged the message, Master [05:18:23] Hi, I am trying to figure out the server/datacenter setup used by wikimedia as people are using it as an example in a discussion [05:19:30] as far as I understood, there are currently 3 datacenters, one in tampa, florida, one in ashburn, virginia, and one in amsterdam. the first two replicate everything and could each run wikipedia completely, while amsterdam is just caching. did I get that right? [07:50:06] JanSch: it's a little more complicated than that [07:50:52] JanSch: there's 2 in tampa, 1 in ashburn, 2 in amsterdam. some work going on getting another one in the US [07:51:43] JanSch: some are primarily used for caching (e.g. amsterdam and the new one for the US that's still being planned) [07:52:50] JanSch: some are (or will be) able to host the entire site for the whole world without help from another datacenter (e.g. ashburn and tampa) [07:54:00] where's our announced trial ipv6 services this weekend? [07:54:19] liangent: in scrollback iirc. but i can't remember for sure which channel [07:55:05] ah, thanks! do you know how the datacenters looked around mid-2010? 1 in tampa running all the DBs etc., 1 in amsterdam caching, possibly 1 in seoul caching? [07:55:11] JanSch: idk how far along ashburn is in being able to do *everything* by itself. and actually right now i think tampa can't do everything by itself either [07:56:08] i mean some services have moved to ashburn while the corresponding ones in tampa are rebuilt from scratch. (on fresh OS installs) [07:56:22] i don't think seoul was around that recently [07:56:31] search the SAL for yaseo [07:56:34] !sal [07:56:34] http://bit.ly/wikisal [07:57:03] !sal del [07:57:04] Unable to find the specified key in db [07:57:08] !sal del [07:57:08] Unable to find the specified key in db [07:57:11] !sal [07:57:12] http://bit.ly/wikisal [07:57:22] petan|wk: ^^ [07:58:10] !sal unalias [07:58:10] Alias removed! [07:58:28] @search wikisal [07:58:28] Results (Found 1): serveradminlog, [07:58:35] !serveradminlog del [07:58:36] Successfully removed serveradminlog [07:58:44] jeremyb: Done :) [07:58:57] hah [07:59:13] !sal is http://wikitech.wikimedia.org/view/Server_admin_log [07:59:14] Key was added [07:59:25] !serveradminlog alias sal [07:59:26] Created new alias for this key [08:00:57] JanSch: the tampa and amsterdam pairs both have very fast links between the datacenters so they can behave sorta like the whole city is a single datacenter [08:01:44] JanSch: people that know all of this better than i should be on to answer some more in the next hour or two. but they will also be distracted [08:01:54] JanSch: can you give some context? link to the discussion? [08:03:44] jeremyb: you don't really want to read the discussion. the german pirate party is running their IT on a budget of 30.000 EUR per year. includes a mediawiki, the etherpad install used by most germans, web sites, synchronized mailinglist/forum/nntp etc. [08:04:28] ok, /me is listening [08:04:45] seems seoul was killed nearly 6 yrs ago [08:04:48] each time something does not work, a lot of people suggest to a) distribute the system among the individual federal states, b) just rent a bunch of root servers for 40 EUR/month with unlimited traffic [08:04:49] based on http://wikitech.wikimedia.org/history/Yaseo_migration_plan [08:05:21] what's a root server? i guess that means dedicated [08:05:32] yeah, german term for physical, dedicated server [08:05:54] so that comes to 2500 EUR/month [08:06:12] do they have to pay VAT? [08:06:40] in which case call it 2000 EUR/month that the provider would get [08:07:18] and is that just hw/network/etc. or is that also including admin/maint/devel ? [08:07:58] "root server" means you get a box with ssh, if the hard disk breaks someone swaps it, and the rest is your job [08:08:52] both of the suggestions come from people without a clue and would not work at all, of course. [08:09:57] the budget is used to rent a rack including power/network, lease some hardware, and do everything else with volunteers. [08:10:27] well nothing you said about the current setup contradicts "just rent a bunch of root servers for 40 EUR/month with unlimited traffic". that might be what's already been done. or dozens of other possibilities [08:10:45] so, admin is volunteers? [08:10:49] oncall? [08:12:01] admins are volunteers, I don't know how physical work on the servers is handled. [08:12:22] anyway, maybe you should look into #puppet or e.g. ##infra-talk ; (wikimedia does use puppet. but in a kinda weird way). /me is interested in following the discussion more though [08:12:33] the thing is, root servers for 40 EUR/month would simply not work at all b/c of traffic limits in the "unlimited" traffic etc. [08:13:03] oh, the admins are competent and they are using puppet and other tools, the point is that 30kEUR is simply not enough to run all the stuff. [08:13:08] ahhh, like US cell phone companies. unlimited is a lie? [08:13:44] yeah. one of those "unlimited" offers is really unlimited - but after x TB you get throttled to 10 MBit/s, except if you pay per TB... [08:13:46] why not? link to the budget? or other info about specs/needs/load? [08:13:56] i could see it running for much less than that or much more [08:14:07] looking for the info [08:14:37] services offered http://wiki.piratenpartei.de/IT/Dienste [08:17:26] ahhh, OTRS [08:17:32] it's slightly outdated, but the fun part is mailserver with 1.000.000 mails/day and mailing lists synced to a web forum, etherpad with ~6000 concurrent users, mediawiki with 22 GB database (uncompressed, compressOld will be run soon), jabber server with 10k concurrent connections... [08:18:54] oh, and of course web site of a political party that rose from 12k to 30k members within the last year. [08:19:42] welp you really should try some other channels (like ##infra-talk). [08:20:55] as I said, the people who run it know what they do, they just need some money for real hardware. and something to fight off the trolls who suggest to rent a few servers or distribute it [08:21:32] "wikipedia ran on one central database/storage datacenter for years" is a good argument for the trolls/idiots [08:21:59] well you could look at past annual reports to see what the tech budget was [08:22:23] and maybe it's even broken down into datacenter vs. other costs [08:23:32] hm, interesting idea, however it is hard to compare wikipedia load (basically one high-load service) to our load (lots of different services). I'll have a look though [08:23:57] anyway, i was thinking about comparisons to wikipedia. one thing that comes to mind is wikipedia's cache hit rate. do you have graphs of cache hits vs. misses? which caches do you use? link to your munin? (anyway, this is getting spammy for this channel...) [08:31:44] is the iwlinks.sql.gz dump relatively new? [08:32:41] lars_: certainly that depends which lang you're speaking of? [08:32:46] it should be dated, no? [08:32:50] or is that one for all langs [08:33:22] the db table was added in MediaWiki 1.17, http://www.mediawiki.org/wiki/Manual:Iwlinks_table [08:33:57] but how long has it been part of the XML/SQL dumps? [08:34:12] where did you get it from? [08:34:25] dumps.wikimedia.org, any of the languages [08:35:25] so, it should be as old or new as the rest of the files for that wiki? [08:35:58] yeah, it should be dated [08:36:07] e.g. http://dumps.wikimedia.org/svwiki/20120525/svwiki-20120525-iwlinks.sql.gz [08:36:29] jeremyb: I'm asking which year and month it was introduced. It seems to have been around since at least April 2011. [08:37:09] ahhh [08:37:19] it's been around for a good long time [08:38:30] good [08:40:50] but it wasn't there in November 2010, http://dumps.wikimedia.org/archive/2010/2010-11/dawiki/20101102/ [08:41:58] Wed Mar 9 12:03:23 2011 [08:42:06] https://svn.wikimedia.org/viewvc/mediawiki?view=revision&revision=83580 [08:43:10] recall that deployment of 1.17 was late feb/early march [08:43:17] so that's about as soon as it could have happened [08:43:19] http://www.mediawiki.org/wiki/MediaWiki_1.17/Wikimedia_deployment [08:47:07] ok thanks [09:42:19] ok, I'm making progress on including interwiki links in my external link stats [10:17:30] !log hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production)' [10:17:35] Logged the message, Master [10:41:52] jeremyb: so zwinger is a dead box [10:41:56] it used to be our bastion host [10:42:00] replaced by fenari nowadays [10:42:11] as for the apache conf. I can sync it apparently [10:42:33] attending a presentation about the Gadgets right now [10:42:37] will do the sync after [10:42:51] sure [10:43:08] hashar: i was just noticing you had a key in admins.pp that mentions zwinger [10:43:23] yup that is just the keyname [10:43:35] I am not sure it has any use, probably just a description [10:44:47] yeah, but usually it indicates where it was generated. unless you change it [10:46:09] k [10:46:12] https://gerrit.wikimedia.org/r/#/admin/projects/operations/apache-config,access [10:46:19] I can't merge apache-config changes [10:46:22] need someone from ops [10:47:34] * Loki eyes jeremyb [10:48:00] * jeremyb catches Loki ? [10:48:26] thought you were someone else [10:48:27] o.o [10:53:05] o_O [11:53:24] how do you purge a result from a search engine manually? http://en.wikipedia.org/w/index.php?search=TORtanic&title=Special%3ASearch [11:53:28] I already deleted the redirect [11:56:02] Thehelpfulone: i think you wait a day? [11:56:41] so there's no way to manually purge it? ok, I'll let 4chan have their fun for another day [11:56:56] hah [11:57:07] i'm sure it's possible [11:58:03] but i think it would require either ops or rainman. and idk if the relevant code/command exists or is documented or would need someone to figure out how to do it [11:58:14] (or someone that already knows offhand) [12:01:54] Thehelpfulone: yes there is no way to force it [12:02:13] because there is no cache to purge [12:02:32] ah :( [12:02:48] I mean there is a cache, but it can't be purged like using wiki [12:04:05] you would need to poke someone from ops and they are kind of hiding from a world atm [12:04:34] hey domas [12:04:34] well you would also need to make a case for it. and i think the case isn't so strong [12:04:51] is there daniel somewhere, domas? [12:05:05] yes. Danny_B|backup. ;P [12:05:21] jeremyb: I mean mutante [12:05:44] oh, not kinzler? ;) [12:05:50] zahn [12:05:51] XD [12:09:24] hey petan|hackaton [12:09:33] hi [12:09:35] do you have sysadmin permission ? [12:09:39] no [12:09:53] maybe if you tell my what is that [12:09:55] petan|hackaton, any chance to find somebody who has ? [12:09:57] I tell you if I have that [12:09:58] :D [12:10:12] you mean shell access right? [12:10:16] petan|hackaton, just some changes to made in a site config, it won't take more than 5 minute [12:10:18] minutes [12:10:34] oh right I can do changes to site config, problem is pulling them to prod [12:10:40] what changes [12:10:51] I already reported them to Bugzilla [12:10:55] i [12:10:56] id [12:11:21] https://bugzilla.wikimedia.org/show_bug.cgi?id=37271 [12:11:29] Quentinv57: next step: move to gerrit. anyone with a gerrit acct can do that [12:11:37] that's what I go do [12:12:00] jeremyb, what ? siteconfig change ? [12:12:11] yes [12:12:28] ok sec [12:12:28] Quentinv57: was that advertised more widely to the community? [12:12:42] Quentinv57: there should be some kind of consensus for that [12:12:55] jeremyb, that's really weird that not only sysadmins can do it... [12:12:57] ahh, i see a link to http://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Sondage/Activation_du_statut_de_cr%C3%A9ateur_de_comptes_utilisateur [12:13:01] config change itself is a matter of few minutes [12:13:02] petan|hackaton, there is, just read the link [12:13:25] config change itself is a matter of few minutes [12:13:26] i can't read it but if all the crats say it says that then... [12:13:35] oh lag [12:13:38] hashar can have a gander if he likes ;) [12:13:55] petan|hackaton: do you want to do the change or should i? [12:14:01] I will do that [12:35:44] Quentinv57: done, https://gerrit.wikimedia.org/r/#/c/9958/1/wmf-config/InitialiseSettings.php needs to be merged by ops now [12:36:05] petan|hackaton, thanks, and who are ops ? [12:36:39] I will poke some, sec [12:41:18] hashar: are you around? can you check thathttps://gerrit.wikimedia.org/r/#/c/9958/1 [12:43:41] petan|hackaton: busy with puppet stuff right now :-D [12:43:54] ok, maybe Reedy can review it [12:44:50] hashar: peekaboo [12:45:11] btw hashar where are you now [12:45:26] there [12:55:59] was there a discussion about the content="" namespaces, now marked so in the API? [12:57:07] e.g. why is ns=6 (File:) not a "content" namespace? [12:57:48] !g 9958 [12:57:48] https://gerrit.wikimedia.org/r/#q,9958,n,z [12:57:58] Krinkle: so yeah somehow there [12:58:04] Krinkle: but too many people to see ;-] [12:59:18] LA2: It's $wgContentNamespaces or something [12:59:32] LA2: By default only the main ns is a content ns, unless otherwise configured [13:02:53] !log midom synchronized wmf-config/db.php [13:02:57] Logged the message, Master [13:03:18] hey RoanKattouw ! :) [13:03:24] do you have 0.5 seconds, please ? [13:03:44] wow, that's short [13:03:50] Quentinv57: Sure! [13:04:11] RoanKattouw, if you can just review the change petan|hackaton made [13:04:16] https://gerrit.wikimedia.org/r/#/c/9958/1 [13:04:30] hahahaha [13:04:37] Quentinv57: it's already merged [13:04:48] hashar did it as you said it [13:04:59] It says 3:04pm so it was merged like a minute ago [13:05:06] Quentinv57: I am working on it [13:05:06] RoanKattouw, oh, sorry, I thought that he was busy [13:05:18] hashar, RoanKattouw : thanks a lot to you two :) [13:05:29] not quite. hashar beat Quentinv57 by 8 secs [13:05:31] Quentinv57: do you happen to be somewhere in Berlin hackaton? [13:05:41] well [13:05:43] technically [13:05:50] petan|hackaton: did the patch. I am just the assistant [13:06:16] hashar, no, I'm not part of hackaton, sorry [13:06:49] domas: You should commit your db.php changes [13:07:48] Quentinv57: brule en enfer!!! :-]]]]]]]]]] [13:07:54] Quentinv57: je m'occupe de dployer le changement [13:08:02] Quentinv57: est tu bureaucrate sur frwiki ? [13:08:11] hashar, t'es fran�ais ? coool :) [13:08:17] non, je ne suis pas bureaucrate sur Wiki fr [13:08:26] mais un bureaucrate m'a demand� de m'en occuper [13:09:23] !log hashar synchronized wmf-config/CommonSettings.php 'Commits: 6bef518 (wgHTCPMulticast only used on production cluster) and 882dd69 (wgLoadScript only used on production) -- was not correctly deployed earlier' [13:09:26] Logged the message, Master [13:09:28] RoanKattouw: can you do anything about 'no space left on device' ? ? mw64 has the issue [13:09:37] Meh, looking [13:10:58] !log hashar synchronized wmf-config/InitialiseSettings.php '59753a9 (allow bureaucrats on frwiki to add+remove accountcreator group' [13:11:02] Logged the message, Master [13:11:21] harriv, youhooo ! it works, thanks a lot :) [13:11:25] ahh! [13:11:32] hashar, I meant to speak to you :) [13:11:40] gooood :) [13:13:50] Quentinv57: "you are welcome" comme on dis [13:16:14] RoanKattouw: I think there was an issue with some debugfs system that just fill up a path in /var/lib something [13:17:43] 6.2G /usr [13:17:44] wtf [13:19:51] ohh [13:19:56] /usr/local/apache [13:20:01] that is cause of the f****** l10n cache [13:20:10] Wait no [13:20:10] several of them since we have 1.20wmf{2,3,4} [13:20:13] isn't it ? [13:20:26] It's because /usr/local/apache isn't on a separate filesystem like it should be [13:20:32] This box didn't get picked up in Peter's migration [13:21:07] maybe it is scheduled for decommissioning / removal [13:21:18] cause I had the same issue on two boxes last week [13:21:24] and Peter just killed them [13:21:54] notpeter: murderer [13:23:19] hah [13:24:55] hey, maybe you could make in the hackaton a sprint to remove the deprecated calls to LogPage::logName ? [13:25:45] I count 14 calls left [13:25:47] If I had a copy of MW locally and an IDE ;) [13:26:01] why do you need an IDE? [13:26:12] makes finding callers a bit simpler [13:26:18] right click, find all usages [13:26:27] oh, that's no problem [13:26:33] what box? [13:26:39] RoanKattouw: ? [13:26:50] notpeter: mw64 [13:27:01] might have been down when I migrated junk [13:27:44] http://pastebin.com/zJvTizaC [13:27:45] yeah, was turned off on may2 [13:28:11] Reedy -> http://pastebin.com/zJvTizaC [13:29:14] RoanKattouw: I'll move the junk around and then you can resync? [13:29:46] Suer [13:29:51] notpeter: Resync is just 'sync-common' [13:30:04] ok, I'll run that on the host when I'm done [13:30:05] as root? [13:30:08] or as me? [13:30:15] As you [13:30:22] kk [14:07:49] !log midom synchronized wmf-config/db.php [14:07:53] Logged the message, Master [14:09:12] RoanKattouw_away: should be back up and cleaned now [14:21:41] notpeter: Looks good thanks [14:21:46] no pro [14:21:47] b [14:44:07] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37294 - Add English Wikibooks as import source at Vietnamese Wikibooks' [14:44:11] Logged the message, Master [14:49:00] !log reedy synchronized wmf-config/CommonSettings.php 'Bug 37211 - Set $wgUseCombinedLoginLink = false' [14:49:04] Logged the message, Master [14:55:05] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 37237 - Change Wikisource namespace for Tamil wikisource' [14:55:10] Logged the message, Master [15:21:04] signing off from Berlin [21:02:08] k [22:57:33] gn8 folks