[01:22:36] Katie Horn 0wns [02:28:42] * jeremyb wonders if there was spme trigger for spage to say that. i think no one was up speaking any more at that point, right? [02:29:57] !log LocalisationUpdate completed (1.20wmf11) at Sat Sep 15 02:29:57 UTC 2012 [02:30:10] Logged the message, Master [02:36:34] @add #wikimedia-otrs [02:45:37] (" ) [05:41:17] whym: sorry for not noticing the CLDR plural news [05:41:29] I made you lose a lot of time I think [05:42:27] but reading the whole https://www.mediawiki.org/wiki/MediaWiki_1.20/wmf11#Core_changes to find such stuff is a bit too much for me [05:52:28] Nemo_bis: sorry for not noticing here too, *I* should have double checked the page! [05:54:36] whym: looking for "plural" on the page would have revealed the commit, but for some reason I assumed the new parser wasn't going to change the actual rules... [05:54:52] and niklas has made an update https://bugzilla.wikimedia.org/show_bug.cgi?id=40251 [05:54:55] yep [05:55:05] I also notified mediawiki-i18n now [05:56:00] of course this all staff meetings in SF are slowing everything down... [06:05:43] Nemo_bis: I remember there is a saying something like "never do a major release before holidays" [06:09:03] hello [06:09:16] This is Butch Bustria from Wikimedia Philippines [06:09:37] We are currently having an outreach project right now on wiki editing [06:09:46] oh, again? [06:09:50] this is really bad timing [06:10:09] you need to give 5-10 days advance warning [06:10:10] Can we request to lift IP address 202.92.128.224 [06:10:18] and 202.92.130.3 [06:10:20] WMPH-Exec8: file a bug [06:10:26] i understood [06:10:35] its spontaneous [06:10:40] i don't think you understand [06:10:44] this is *really* bad timing [06:11:07] in fact i think you're not likely to get it fixed in the next 12 hrs [06:11:15] do you need it after 12 hrs from now? [06:11:25] anyway, file a bug [06:11:50] and tell us the bug # of course [06:12:56] seems to be 14:12 there now [06:13:03] is this a multi-day event? [06:14:36] ok, i'm off. back in 7-8 hrs [06:25:18] Bug 40263 - Request to lift IP address for mass creation - Philippines SFD 2012 [06:25:26] from now until 15:00 UTC [06:26:22] We appreciate your kind action. [06:26:27] WMPH-Exec8: you have no sysop at the event? [06:27:22] no sysop [06:28:57] WMPH-Exec8: en or tl? [06:29:31] both. sorry. wikipedia.org only [06:32:29] WMPH-Exec8: what's your username? [06:32:56] Exec8 [06:33:27] ok [06:33:31] oh, only 24 persons [06:33:35] https://www.mediawiki.org/wiki/Help:Mass_account_creation is for you [06:33:36] yes [06:33:54] you don't even need noratelimit I think [06:34:33] Nemo_bis: i don't follow? [06:35:16] Nemo_bis: tlwiki has no ability for crats to grant accountcreator and there are no current accouncreators [06:35:43] ok [06:35:44] Nemo_bis: easiest is to get an enwiki crat to grant it (since they're covering enwiki anyway) and then use that account [06:35:56] hm [06:36:23] en.wiki has a rule requiring identification for account creators [06:36:30] that must be new [06:36:45] I think they're using an OTRS queue or something [06:36:46] Nemo_bis: frankly i have no problem with just turning down this request outright. too little notice [06:36:58] do we need any intervention from any tl admins? [06:37:02] maybe a temp access won't need it [06:37:05] WMPH-Exec8: no [06:37:06] no, there's a toolserver app. ACC. at least if it's still in use [06:37:15] WMPH-Exec8: please write on the bug which wikis are affected [06:37:22] jeremyb: yes but where does data go? [06:37:33] Nemo_bis: toolserver of course [06:37:40] * Nemo_bis shrugs [06:37:49] is there any not too bureaucratic bureaucrat around? :) [06:37:56] WMPH-Exec8: you need a tl or en admin to show up at the event in person [06:38:08] * jeremyb runs away [06:39:00] Bluemask , jojit_fb and seav of TL are here [06:39:35] they are TL admins [06:39:43] wow [06:39:48] wowwww [06:39:51] easy tehn [06:39:56] just follow https://www.mediawiki.org/wiki/Help:Mass_account_creation [06:40:12] 15 06:26:27 < jeremyb> WMPH-Exec8: you have no sysop at the event? [06:40:12] 15 06:27:22 < WMPH-Exec8> no sysop [06:40:16] been to that site [06:40:26] i'm going to stab someone... [06:40:28] at our last WMIT event I had a tablet logged in with a sysop account borrowed from another person and went around creating accounts :p [06:40:43] seav is a bureaucrat [06:41:41] admin is enough [06:42:27] What's up? [06:43:32] aww superpowers coming [06:45:30] Reedy: account creation at a event with no crat [06:46:55] * p858snake|l bribes Reedy with stropwaffles for https://bugzilla.wikimedia.org/40263 [06:51:26] p858snake|l: they have a crat and 2 sysops! [06:51:30] and 2 IPs even [06:51:41] as we say, "grasso che cola" [06:59:53] does this make any sense? https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=blob;f=wmf-config/InitialiseSettings.php;hb=7f319eb18c36ee09effaa75058515d03c9c2cb3d#l6370 [07:03:52] we are in a computer laboratory and may prefer do the account creation themselves [07:04:35] they can [07:04:57] you only have to login in advance (and check they don't make disasters), did you check the help page? [08:42:50] does someone know when the new planet will be enabled? [08:43:25] haven't heard anything about it lately [08:43:28] the current planet is lacking several posts, like Wiki Loves Monuments in French, because feeds have been added only to the new planet config [08:43:31] https://meta.wikimedia.org/wiki/Planet_Wikimedia#Requests_for_inclusion [08:43:43] Should be out soon we're rolling out a new version of planet at the moment and it will be up when that's done (probably later today) Jalexander (talk) 19:30, 22 August 2012 (UTC) [08:43:51] the new system (already live here ) which should happen in the next day or two. Jamesofur (talk) 23:39, 26 August 2012 (UTC) [08:45:26] well slap some folks around :D [08:45:52] just bite a few thumbs and chuck a couple of gloves onto the ground? [08:46:47] Nemo_bis: or blog about it on a blog that is still on the current planet [08:47:14] or beg thedj to help [08:47:21] https://meta.wikimedia.org/w/index.php?title=Planet_Wikimedia&diff=4130472&oldid=4114918 [08:47:41] it's like four lines of copy and paste, come on :p [08:48:19] i'm not sure how the current config is running... [08:48:38] thedj: it's just a file on svn [08:48:38] it's still on svn ? [08:48:48] I think so [08:49:01] lemme switch computers [08:49:13] like https://www.mediawiki.org/wiki/Special:Code/MediaWiki/115645 [08:49:33] I suppose (and hope) they didn't do double work, made the new puppet config only for the new syste, [08:51:38] too bad: http://svn.wikimedia.org/viewvc/wikimedia/trunk/tools/planet [08:51:41] no longer exists [08:52:53] !log someone should deploy Planets update, or add blogs to the old config because otherwise WLM has passed before the blogs are added to the stream. [08:53:04] Logged the message, Master [08:53:44] aww [08:53:51] thanks for trying, thedj [08:54:36] that's not what I should use that for, but it's not like filing a bugzilla ticket ever results into someone noticing it. [08:54:55] and i'm not around 24/7 to poke with sticks either :D [08:55:14] indeed [08:56:02] so now where does the old config live, one wonders? [08:56:35] possibly only on disk atm... [09:40:26] Nemo_bis: actually it still exists. it's in mediawiki, not wikimedia. But it doesn't seem like i'm still allowed to commit there. [09:41:26] svn: E175013: Access to '/svnroot/mediawiki/!svn/act/f45d5d6d-8546-4181-ab80-929f3dbd6667' forbidden [09:45:11] thedj: sigh [09:47:50] posted on the ml. [09:48:03] gtg now [09:48:14] Just added throttle.php to noc conf ;) [09:48:43] Right... [09:56:38] !log reedy synchronized wmf-config/throttle.php 'Bug 40263 - Request to lift IP address for mass creation - Philippines SFD 2012' [09:56:47] Logged the message, Master [10:02:31] Reedy: are you also one of the "someone" who could update the old planet's config per DJ's request on wikitech? :D [10:18:16] Nemo_bis: need root for that one, I think [10:18:45] You can likely get mutante to do it on Monday [10:20:46] Reedy: what? you don't have root access on that machine? :-O /me surprised [10:21:00] I has no root anywhere [10:21:07] oh [10:21:13] * Nemo_bis facepalms [10:21:45] singer? [10:23:10] loooks so [10:24:42] yeah, can't even log into it seemingly [11:54:11] !log andrew synchronized wmf-config/throttle.php 'Fix undeclared variable error' [11:54:20] Logged the message, Master [13:28:27] Squid down, thanks. [13:28:36] Hi, I´m getting a Wikimedia foundation error here [13:28:51] oh... [13:29:14] wikimedia error... but I guess I'm not the only one [13:29:21] yeah, down here. [13:29:42] +1 [13:29:43] facepalms, its a Saturday zzz [13:30:55] haha, the french team arrived [13:31:04] Yes :) [13:31:19] Hi everybody [13:31:46] Hi Superjuju10 ! [13:31:50] x) [13:31:53] Superjuju10: Welcome here :D [13:31:57] Hi Jules78120 :D [13:32:01] Woozz: Hi too ! [13:32:20] Hi Woozz Superjuju10 and Jules78120 :) Where do you come from :-P [13:32:47] elfabixx: Tours, in France :) [13:33:24] xquirky servers :-/ [13:33:27] elfabixx: I think I know you [13:33:57] Jules78120: you sure ? [13:34:02] Jules78120: Superjuju10 elfabixx : it's working again [13:34:03] I live in Prugny in France ;) [13:34:20] Woozz: ya, i noticed it [13:34:41] on and off and on and off ... [13:34:49] elfabixx: Hum... Not sure but... [13:34:58] Superjuju10: Oh ! [13:35:02] flood [13:35:10] Superjuju10: We live in the same country ! [13:35:23] Yep [13:35:25] :O [13:35:28] Ow ! that so strange ! [13:35:34] And you, Woozz, where do you live ? [13:35:44] Superjuju10: Do you work in french wiki ? [13:35:47] -_- [13:36:01] Request: GET http://en.wikipedia.org/wiki/Weighing_scale, from 10.64.0.131 via cp1014.eqiad.wmnet (squid/2.7.STABLE9) to () [13:36:03] Error: ERR_CANNOT_FORWARD, errno (11) Resource temporarily unavailable at Sat, 15 Sep 2012 13:34:30 GMT [13:36:21] Jules78120: Yes I'm [13:36:21] fabriceferrer: Hello :) [13:36:31] Sj [13:36:31] Hello fabriceferrer the troll ;) [13:36:43] ... [13:37:19] Superjuju10: What's your real name ? Jules ? [13:37:29] fyi, we're looking to see why memcache has gone crazy and trying to fix the problem [13:37:30] No, my name is Julien ;) [13:37:46] Superjuju10: Okay ! [13:38:05] Superjuju10: Do you want to be my friend ? [13:38:17] Well.... perhaps should we stop, uh ? :) [13:38:21] thx LeslieCarr, appreciated [13:38:30] +1 [13:38:30] Jules78120: Yes :) [13:38:42] Thank you Superjuju10 :-* [13:38:50] You'r Welcom :-* [13:39:09] elfabixx: you're too serious ^^ [13:40:20] what's wrong with you guys ?? x) [13:41:01] elfabixx: do you know Superjuju10 ? He's a nice guy ! [13:42:37] nothing is wrong, Woozz, the french like to practice the art of proper conversation under all circumstances :) [13:44:03] * Superjuju10 confirm [13:44:06] Jan_eissfeldt: we're... crazy ! ;) [13:44:15] hey, is it up already? [13:44:20] Very versy crazy ;) [13:44:23] *very [13:44:54] Jan_eissfeldt: especially me and Superjuju10 [13:45:34] ;) [13:48:53] We are alone Jules78120, we are freedom ! :) [13:49:22] Superjuju10: :D [13:50:24] /wc [13:51:35] bye bye all ! [13:51:47] and happy birthday Superjuju10 ! [13:56:36] hi [13:56:54] Juandev: hi [13:58:28] Some idea, Exactly whats happend? [13:59:26] wilfredor: tropezaron con un cable y bueno, el resultado es el que ves :P [13:59:56] mafk: I see [14:11:52] RD: thanks for the reminder, i should have breakfast [14:12:03] aka unhatched chickens [14:12:07] w/ spinach [14:12:12] Sounds good [14:12:13] er [14:12:20] not the last bit [14:12:23] hah [14:12:30] I'll go without the spinach, more chickens. [14:14:52] thanks LeslieCarr, a lot better [14:16:56] sDrewth: it's still mostly dead I think? [14:17:04] moin odder [14:17:10] moin jeremyb [14:18:36] Hydriz: see what you did? [14:18:59] what? [14:19:12] you broke it again because you said it was fixed to early [14:19:38] it loads zzz [15:07:35] werdna: hey [15:07:48] liangent: just finished writing you a mail [15:07:55] we've been having some downtime [15:08:03] and that request that AdvancedSiteNotices makes is uncached [15:08:54] I remember we tested it and made sure it's cached during development [15:08:59] but I can't remember details [15:09:04] because main dev is not me [15:09:13] Well, the response has Cache-Control: maxage=0 [15:09:22] Or maybe private [15:09:26] I don't remember which [15:09:32] but in either case it is not cached by squids [15:15:47] werdna: I guess it became uncached because of https://bugzilla.wikimedia.org/show_bug.cgi?id=36142 [15:16:06] mark: ^^ [15:16:30] looking [15:17:01] hmmph [15:17:10] doesn't look related [15:17:18] liangent: but it's not diff pages in question [15:18:13] werdna: all un-auto-purgable pages are excluded from caching in the fix [15:18:26] mm I can see that now [15:19:24] werdna: so is there a param for index.php to make it set smaxage? [15:19:31] like the one for api.php [15:19:31] I don't know [15:19:37] where is the /cn.js call coming from? [15:22:01] werdna: anyway I can turn it to use api and set smaxage I think [15:22:15] liangent: can we wait for now? [15:22:26] just give us time to exclude it as the source of the site issues [15:23:01] werdna: ok [15:23:07] thanks [15:25:25] werdna: what do you suggest for the (s)maxage value I should use? [15:25:45] not sure… I think mark / LeslieCarr would have a better idea [15:27:06] and we will of course say: as long as you can make it ;) [15:27:34] hah [15:27:45] took me a few secs to parse that one [15:28:25] is 1hr ok? [15:31:23] liangent: i think outage first, your notice later [15:32:40] liangent: what's the notice about anyway? [15:33:00] jeremyb: replacement of MediaWiki:Sitenotice [15:33:28] huh? you have a notice about a notice? [15:33:44] jeremyb: I don't understand your question [15:33:57] iirc the first time it came out because MediaWiki:Sitenotice doesn't handle zh-hans/zh-hant well [15:34:26] what notice is supposed to show up at the top of the page right now? [15:34:36] is it actually a big deal that it's not shown? [15:35:20] jeremyb: not really I think [15:35:28] but I want to fix it [15:35:39] liangent: so what is it supposed to show now? [15:35:44] liangent: what's the message? [15:35:55] with that script disabled, nothing to show then [15:36:09] ... [15:36:24] i can just repeat myself i guess [15:36:34] and when it's not disabled? [15:38:16] jeremyb: current article writing activity, and some "ads" target specific people [15:38:20] eg local meetups [15:38:34] ok [15:40:14] werdna: so I fixed the script to use api.php with smaxage, and I'm going to remove your "return" so I can test it myself? [15:40:21] anyway it's still non-default in gadgets-definition [15:40:28] Can you do it tomorrow? [15:46:34] werdna: then I added these http://zh.wikipedia.org/w/index.php?title=MediaWiki:Gadget-AdvancedSiteNotices.js&diff=22808369&oldid=22808177 to allow debugging with local hack [15:48:55] i don't see how that could possibly help [15:50:02] jeremyb: in firebug I set a breakpoint at 'if ( _return )' and edit _return to false then continue [15:50:18] uhuh [15:50:33] can't you just have it check a GET param? [15:51:31] jeremyb: it needs more code and when debugging I have to use firebug [15:52:08] ok, if it makes you happy [15:52:10] ;) [15:55:18] how about not even tomorrow but next week when people are working ;) [15:55:34] i'm gonna get breakfast now [15:56:14] liangent: ^^. so monday or tuesday [15:56:59] eh every time I see your code deployment late by one day [15:57:16] because you're utc-?? and I'm utc+8 [15:57:26] so should be Tuesday or Wednesday for me then? [15:58:00] btw it's now using this url [15:58:00] http://zh.wikipedia.org/w/api.php?page=Template%3AAdvancedSiteNotices%2Fajax&variant=zh&action=parse&format=json&maxage=3600&smaxage=3600 [15:58:21] but it keeps returning MISS from X-Cache-Lookup and X-Cache [15:58:26] so what's wrong? [16:00:30] i got a hit [16:01:04] and age=38 [16:07:33] jeremyb: ok if you can confirm this works, I'll leave it as its current state and wait for Monday [16:07:47] liangent: well what are you getting? [16:08:58] jeremyb: X-Cache: MISS from cp1004.eqiad.wmnet, MISS from cp1018.eqiad.wmnet [16:08:58] X-Cache-Lookup: MISS from cp1004.eqiad.wmnet:3128, MISS from cp1018.eqiad.wmnet:80 [16:09:29] and how are you checking? [16:47:42] jeremyb: I load that url in browser and check firebug console [16:48:01] how are you loading it in browser? [16:49:47] by clicking reload or locating cursor in location bar then enter [16:49:49] neither works [16:50:41] well it's surely just cached in your browser's local cache and not making a network request at all [16:50:51] clear your cache and then location bar + enter [16:50:54] don't hit refresh [16:53:08] jeremyb: still miss [16:53:28] and I'm sure it's not cached locally because everytime it gives a new "date" header [16:55:04] liangent: errrr, i want to watch you do this... [16:55:09] liangent: can you tcpdump it? [16:55:16] liangent: are you logged in??? [16:55:42] liangent: you have to do it when not logged in. clear your wikipedia.org cookies and try again [16:56:21] I pressed "log out" but didn't clear cookies [16:56:29] clear em [16:58:12] jeremyb: ok finally HIT now [16:58:16] which cookie is annoying? [16:59:05] idk [17:27:49] hey [17:38:40] * Hazard-SJ knew something was wrong [17:42:43] hello ? [17:42:45] anyone ? [17:42:53] it's saturday... [17:43:02] I'm struggling to find anyone active [17:43:23] hi [17:43:57] hey paravoid [17:44:03] paravoid: you familiar with mediawiki extensions ? [17:44:09] jeremyb: the process is usually: Check if guillom is around, and if not, ping someone else on the team :) [17:44:24] average_drifter: we're in an emergency situation right now ... [17:44:29] LeslieCarr: oh ? why ? [17:44:49] because the site is falling over [17:44:53] average_drifter: try #mediawiki [17:44:58] average_drifter: see topic. ;) [17:45:02] guillom: heh [17:45:16] LeslieCarr: can I know more ? [17:45:26] not from LeslieCarr [17:45:30] alright [17:45:31] other people can tell you about it [17:46:19] average_drifter: there was an outage this morning about 4ish hrs ago [17:46:38] average_drifter: everything recovered and looked good. and now maybe it's back [17:46:49] average_drifter: or at least there's some current major problem [17:46:57] average_drifter: take a look at status.wikimedia.org [17:48:36] jeremyb: hum, interesting that the outage of the servers is so severe, that it even affects the nagios load ... [17:48:56] T3rminat0r: nah, maybe that was already yellow to begin with [17:49:10] T3rminat0r: but nagios does sometimes have problems [17:50:01] seeing the responsetimes rising again in the nagios graph on status. ... I guess that is because it has to do lots and lots of repeat-checks due to downed servers [17:52:39] jeremyb: can I ask you about the clicktracking extension ? [17:52:54] average_drifter: nope. you should ask on monday [17:53:08] ok [17:57:10] * Hazard-SJ is annoyed that vandals are doing there things and he can barely do anything [17:58:10] My cookie keeps getting eaten after the long wait to load the pages :'( [17:58:31] Hazard-SJ: yea, as the topic states, there are server issues... [17:59:28] T3rminat0r: I was aware of it, but the sad thing is that vandals are still at work. [17:59:52] ofcourse ... understand your problem ... but well, it will get fixed eventually ... :S [18:07:11] jeremyb: do you think the problems might be caused by the mediawiki update to 1.20wmf11 ? [18:07:45] i don't personally. if you made me guess [18:07:51] but i don't really know [18:08:21] ok [18:08:36] yea, I was interested in your opinion :) [18:10:49] Just FYI, I am getting this error message when I try to open a specific article. [18:11:13] don't do that [18:11:28] I can open other most articles without any problem. [18:11:30] T_suzu: please don't use /notice [18:11:32] T_suzu: 1) we know there's an outage, see the /topic [18:11:55] got it [18:12:08] T_suzu: 2) don't paste more than 1-2 lines at a time. use a pastebin or figure out a way to shorten/abbreviate it [18:12:28] T_suzu: 3) as you just heard from DeltaQuad, no /notice [18:12:35] danke ;) [18:13:11] jeremyb: will be more cautious next time. [18:13:34] T_suzu: the main problem is the /notice but also the pasting [18:13:50] T_suzu: telling us something we already know is not so horrible [18:14:10] makes some users machines go beep [18:17:27] My apologies if anyone had to climb out of bed [18:18:00] * jeremyb runs off to the WTM event [18:38:11] tstarling cleared profiling data [18:44:45] Wow, the sites are slow this eve. :-( [18:45:01] intentionally so [18:45:06] even [18:45:38] what's gone wrong? [18:45:52] no idea, but things are messed up [18:46:07] not intentionally certainly [18:46:35] !log tstarling synchronized php-1.20wmf11/includes/objectcache/MemcachedClient.php 'memcached profiling' [18:46:39] didn't you reduce maximal requests per client? [18:46:40] tstarling cleared profiling data [18:46:44] Logged the message, Master [18:47:06] i think you've misread [18:47:25] let me consult the docs [18:47:27] i've only heard [18:49:47] giftpflanze: http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild [18:49:56] that's what was actually changed from 300 to 4 [18:50:14] ah, ok [18:50:22] ok, then the missinformation is on my bad, sorry [18:51:30] T3rminat0r: no, wasn't you. i think it was an unnamed someone else ;) [18:51:32] that's scary technical jargon … activities in a child lifetime, after which they die? :-/ [18:51:55] really? [18:51:56] hi, what is with servers? WMF's sites loads very slowly, mobile version doesn't work... [18:51:57] jeremyb: I told giftpflanze about the log entry [18:52:15] T3rminat0r: and the log entry was wrong [18:52:27] Base: known issue [18:52:35] Base: see topic and http://status.wikimedia.org [18:53:23] jeremyb: yea, from that point of view, the fault lies with a certain someone who's stressed enough as it is, seeing what's going on ;) [18:53:30] T3rminat0r: that page doesn't really help, even for those that are tech-savvy. :-/ [18:53:41] !log tstarling synchronized wmf-config/mc.php 'swap out srv237 for srv249' [18:53:43] when all will be stable ok? there are some problems almost all time for 3+ month... [18:53:46] mpeel: green is good, yellow and red are not ;) [18:53:50] Logged the message, Master [18:53:55] T3rminat0r: yes, I figured that. :P [18:54:35] mpeel: it's that simple ... if there's a red dot next to a service (like w:de for wikipedia de), that means there are huge, known problems with that service [18:54:58] point is, if it's red or yellow there, the admins know about it ;) [18:55:02] but I like to know "server N has had kittens, and has gone offline" rather than 'Wiki Platform " has a service disruption... [18:55:53] particularly when I'm working on commons, which doesn't seem to come under a 'Wiki Platform'... [18:57:38] T3rminat0r: sorry, my point here is that the notices should do their best to make it clear what the best-known source of the problem is… not sure they're doing that here. :-/ [18:57:49] mpeel: http://status.wikimedia.org/8777/163404/Wiki-commons-%28s4%29 [18:57:50] T3rminat0r: http://wikitech.wikimedia.org/index.php?title=Server_admin_log&diff=51550&oldid=51549 [18:58:17] jeremyb ;) [18:58:44] T3rminat0r: that's nice, I guess it's just sleeping on the job then. :-/ [18:58:48] mpeel: if they knew what the problem was, the solution would be just a small step ... normally [18:59:44] T3rminat0r: sure, I was trying to draw a line between knowing where the problem lies, and what the problem was. But I'm not meaning to argue here, just to understand. :-) [19:01:52] well, from what I gathered, there is some problem with the caching servers and some other problem on the apache servers [19:02:14] (I doubt this will help in any way, but as you asked ... ;) ) [19:02:32] thanks T3rminat0r. :-) [19:02:41] np [19:02:48] :) [19:04:00] !log tstarling synchronized wmf-config/mc.php 'swapping out other slow servers' [19:04:09] Logged the message, Master [19:04:10] tstarling cleared profiling data [19:07:25] ukwiki is one of projects with maximum problems, isn't it? why so? [19:11:27] Wikis are speeding up :D [19:12:33] they are indeed. :-) [19:14:26] Base: well, the bigger the project, the bigger the problems ;) (at least this time) [19:20:01] ah when WMF will start buying normal servers, but not organising contests like WLM to spend money for nothing..?.. [19:20:37] Base: er, wtf are you talking about? [19:21:23] jeremyb: that WMF should spend money better way... [19:21:40] Base: details? [19:21:52] Base: what should they spend money on that they haven't? [19:22:26] jeremyb: problems with servers for 3+ month is not ok. [19:22:42] [citation needed] [19:22:47] which 3 months? what problems? [19:23:47] jeremyb: you not see problems? I cant help you - i'm not ophthalmologist [19:24:03] Base: ok, i'll just ignore your baseless nonsense then [19:24:52] jeremyb: it is not nonsense. It is unnormal when servers has lag every almost 2 days... [19:25:07] [citation needed]... [19:25:35] jeremyb: just look at logs if you cant see it directly [19:25:50] no, you point out specific parts of the logs [19:28:17] i'll make sure to get right on buying regular servers. i just thought that the mac mini's we buy would work so well [19:31:15] Base: no response? or you're still looking? [19:31:40] jeremyb: what you are waiting from me? [19:32:27] Base: cite specific parts of individual logs if you claim that the logs indicate some recurring lag. what do you mean by "lag" even ? [19:33:54] but your claims so far are really outlandish and not grounded in reality and unsubstantiated. (not to mention unclear: what is "lag"?). so i will default to assuming your words are nonsense for the forseeable future [19:37:42] Base: WMF is not organising contents like WLM, that's done by the chapters. [19:37:55] that's irrelevant [19:38:08] just few minutes ago was serios lags on ukwiki (and not there only) what was displayed at http://status.wikimedia.org . Problems repeats many time during last monthes — what more you need. I'm sure that it proof that current hardware of WMF is weak [19:38:11] he's not stated one technical thing that more money should be spent on [19:38:42] Base: again, define lag [19:38:42] mpeel: it spends money for grants for local wikimedias [19:38:58] jeremyb: (facepalm) [19:39:27] Base: ok, i'll just ignore you ;-) [19:39:44] Base: yes, but it spends significantly more on its tech department. [19:40:02] jeremyb: if it is best that you can do — ok. [19:41:03] Base: i asked you for problems and all you gave me was that something was broken during an outage. it was an outage damnit! wasn't just overload. wasn't something that throwing more servers at would fix [19:42:14] mpeel: but it is seems that not enough [19:43:21] jeremyb: such outages repeats very frequently. [19:43:27] prove it! [19:43:39] jeremyb: ... what. they do. [19:43:53] MatmaRex: ? [19:44:23] Base: we have no indications that we're lacking servers (either the outage at hand or in general) [19:45:39] i'm not saying you guys need to buy more servers [19:45:50] (facepalm) lang:ru:"адресованная другу ходит песенка по кругу" [19:45:58] but various outages and problems keep happening all the time :( [19:46:09] okay, not all the time, but too often [19:46:27] more often than at other similarly sized sites [19:46:36] that's not true [19:47:08] more often than they should, then? [19:47:20] well, sites "should" never fail, but they do [19:48:05] we can make it never fail, but then we can also not ever change anything anymore [19:48:11] I guess that's not what the community wants either ;) [19:51:07] all changes should be tested '''before''' doing it on main servers... [19:51:31] i don't think you understand what is involved here ;) [19:52:59] mark: Ok, but I dont care for your opinion about my understandibleness [19:53:24] you're funny [19:55:14] I concede the point that our site sometimes has less uptime than other major sites (i'm thinking of google) , however we have a much smaller team, infrastructure, and budget. Besides, even with our limitations, we have more uptime than twitter (who has about 20x the tech department and even more than that of budget) [19:55:44] hehe I don't even consider google "similarly sized" [19:56:37] but we definitely don't have a lot of downtime [19:56:37] the only major site i know of that i find down more often than Wikipedia is reddit :P [19:57:40] except that actual uptime statistics published by the companies themselves disagree with wikipedia being down all the time [19:58:25] which companies? [19:58:44] like twitter has its development blog [19:58:51] i dont know if reddit publishes its monthly uptime [19:58:53] (looking now) [19:59:38] i thought you were talking about companies that published stats on wikipedia uptime [20:00:07] i dont think reddit does [20:00:09] (not sure if there are any...) [20:00:17] i'm not referring to any stats, but personal experience [20:00:28] reddit certainly does post-mortems on big downtimes, but i think that's it [20:01:20] hrm, don't see any uptime or monitoring published by reddit [20:01:37] doesn't mean it doesn't exist, just that my search-fu is not up for the challenge [20:06:31] ah, done with the internets, at least for now ;) [23:09:30] So, Gerrit exposes a public JSON-RPC interface which I'm all set to query a large number of times in order to generate some stats [23:09:52] e.g. median code review time [23:10:18] Anyone got any thoughts on a reasonable throttle for that? [23:11:31] (It's pretty much the same performance wise as loading a page on gerrit.wikimedia.org, I would guess, just without the pretty frontend) [23:12:28] jarry1250: Usually 1 request at a time and a few ms between shouldn't heavily affect a service in thise size (just my 2ct) [23:15:50] hoo: Yeah, I was thinking 10000 requests initially over 8 hours, which would make for a 2000ms wait between each, I think that would be enough but I don't know an awful lot about performance issues, especially for Gerrit and its servers [23:19:52] I'm not firm with gerrit either, I just said, that that's usually a good maximum [23:20:20] less for small sites of course and more only if you know what you're doing [23:22:52] just make sure you have a decent useragent set, then play it by ear [23:23:10] you will soon find out if you crash the box or not :p [23:25:03] Hmm, well since I'm going to leave it overnight, I'll go with 5000 requests and a 4 second wait in between to be on the safe side, I think. [23:25:57] Better safe than sorry.