[01:21:08] !log reedy synchronized closed.dblist 'Closing wikimania2011wiki' [01:21:13] Logged the message, Master [01:21:46] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36571 - Please lock wikimania2011 wiki' [01:21:49] Logged the message, Master [01:23:56] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia' [01:23:59] Logged the message, Master [01:25:07] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36595 - Please enable Extention:NewUserMessage on ml.wikipedia' [01:25:10] Logged the message, Master [01:27:28] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36533 - Set sitename to Telugu Wiktionary' [01:27:31] Logged the message, Master [01:30:41] !log reedy synchronized wmf-config/InitialiseSettings.php 'wgShowUpdatedMarker enabled on anything that isn't enwiki or dewiki' [01:30:44] Logged the message, Master [01:36:56] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36663 - Please allow bureaucrats to add and remove autoreviewer status on pt.wiki' [01:36:59] Logged the message, Master [01:39:40] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36522 - Upload link should lead to UploadWizard instead of commons:Special:Upload' [01:39:43] Logged the message, Master [01:42:05] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 36506 - Site logo for Tsonga Wikipedia -- ts.wikipedia.org' [01:42:09] Logged the message, Master [01:53:08] got a question regarding mediawiki and redirects leading to a subsection: Is it "wanted behaviour", that when I have a Redirect called "Site_A", leading to "Site_B#Something", that the URL in my browser reads "Site_A#Something" after I clicked a Link to A? [01:54:01] Example from de.Wikpedia: http://de.wikipedia.org/wiki/Bizarre_Love_Triangle Leads to http://de.wikipedia.org/wiki/New_Order#Singles, but the URL in the browser reads http://de.wikipedia.org/wiki/Bizarre_Love_Triangle#Singles [01:56:45] (figured, I would ask this here, since it's a somewhat technical problem ... I'd guess it has to do with the Redir being done via an AJax call or something ... [01:58:34] yes that is desired [01:59:19] ok [02:00:10] took me some 2 times back and forth till I figured, why a link to page a leads me to A#some, till I saw I wasn't on page a at all ... ^^ [02:13:10] !log LocalisationUpdate completed (1.20wmf2) at Wed May 9 02:13:10 UTC 2012 [02:13:14] Logged the message, Master [02:19:33] !log Running cleanupUploadStash.php over all wikis [02:19:36] Logged the message, Master [02:37:02] !log LocalisationUpdate completed (1.20wmf1) at Wed May 9 02:37:02 UTC 2012 [02:37:05] Logged the message, Master [02:44:05] Can I please get some help at https://secure.wikimedia.org/wikipedia/meta/wiki/Meta:Requests_for_help_from_a_sysop_or_bureaucrat#Lost_Password_for_DieWeisseRose ? [02:44:27] Please don't use secure.wm.o [02:44:38] use https://meta.wikimedia.org [02:44:56] I don't understand. [02:45:09] secure.wikimedia.org is deprecated [02:45:09] Okay, now I understand [02:45:32] I will repost the request to meta [02:47:38] Just checked meta and my request is arleady there. [02:48:32] yeah [02:48:35] It's the same site underneath [02:48:43] It's just the way you linked is the old secure method [02:49:36] Shall I link using meta.wikimedia.org or is it unnecessary? [02:50:51] Shall I repost the text of my help request here? [02:51:44] Per the replies, you've got to prove somehow it's your account [02:55:03] Okay, well I have the text and exact date/time of the watchlist token I set last year and I have access to the email address used by Wikimedia to contact me about the Referendum last August. [02:55:26] hmm [02:55:32] If you got contacted, you must have an email somewhere.. [02:55:43] *an email set somewhere [02:57:47] I linked all of the accounts last year, I'm not sure of the terminology but they was some procdure for setting one password for all the various wiki accounts. [03:00:46] Yeah [03:00:56] So is it just your meta account? [03:04:38] I can't access any of my accounts and when I use my username to request a password, I get a message that there is no assocaited email address but I got the referendum message months after I had the password loss. [03:06:11] Which was your original wiki? [03:06:29] You've got an authed email, but no address set [03:07:23] Okay how can I set that email when I can't access the accounts? [03:08:03] You've got an email set on commons [03:08:23] And on enwikiquote [03:08:43] Try using one of those wikis to reset your password [03:10:13] I can set that email to your global account if needed [03:11:16] I just tried both of them but I'll try again. Can you tell me the username for en.wikiquote. When I tried to reset my password I got: "There is no e-mail address recorded for user 'DieWeisseRose' " [03:12:01] https://en.wikipedia.org/w/index.php?title=Special%3ACentralAuth&target=DieWeisseRose [03:12:04] It's the same username [03:12:51] One set on eswiki too [03:13:41] I've just set that email to your global account [03:15:47] Okay, thank you. I just tried a passwd reset via the URL you gave but got "Action throttled". I guess I'll have to wait. Do you what the reset time is ? [03:15:53] a few minutes [03:16:00] which wiki? I can probably override that [03:16:04] <10? [03:16:25] en wikipedia [03:19:15] I just invalidated your user cache [03:19:28] though I think the throttle is seperate in memcached [03:20:23] shall i try another reset then? [03:24:49] well, the reset time is apparently more than 7 minutes because I just another reset attempt and it's still too soon [03:31:36] 5 in 5 minutes apparently [03:31:39] or default is [03:41:12] I just tried again--no go. [03:41:53] would it make anydifference if I tried a different wiki? [04:16:51] Reedy, I just tried password reset but still get "There is no e-mail address recorded for user 'DieWeisseRose'." [07:07:08] Reedy, thanks for update markers; people on en.wiki are complaining about weird behaviou of watchlist without it, but hopefully it won't last long :) [07:27:47] jeremyb, you edit conflict my emails! [07:28:57] {{fact}} [08:03:42] DieWeisseRose2 Hi! see http://viaf.org/viaf/131008281/#Weisse_Rose_%28Resistance_group%29 [08:07:01] gangleri: DieWeisseRose2 ain't here [08:07:32] thanks [08:23:56] hi gangleri [08:24:10] was that JS on test.wiki fixed? [08:25:51] Hi! Nemo_bis hi! please note that an improved version of http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/viaf/index.html is available - this is a [[authority control]] tool [10:10:15] Hello, Houston, we have a problem ! [10:10:25] Try to edit the last section of the article Montreal. I can't. [10:10:34] http://en.wikipedia.org/w/index.php?title=Montreal&action=edit§ion=21 for instance [10:10:49] I get an error : [10:10:50] Request: POST http://en.wikipedia.org/w/index.php?title=Montreal&action=submit, from 91.198.174.41 via sq78.wikimedia.org (squid/2.7.STABLE9) to 10.2.1.1 (10.2.1.1) Error: ERR_ZERO_SIZE_OBJECT, errno [No Error] at Wed, 09 May 2012 10:06:12 GMT [10:12:13] other pages? [10:12:22] or (as far as you know) just that one? [10:15:21] Not all articles are broken. [10:16:40] very slow saving here [10:16:53] Request: POST http://en.wikipedia.org/w/index.php?title=Montreal&action=submit, from 208.80.152.75 via sq78.wikimedia.org (squid/2.7.STABLE9) to 10.2.1.1 (10.2.1.1) [10:16:56] Error: ERR_ZERO_SIZE_OBJECT, errno [No Error] at Wed, 09 May 2012 10:16:44 GMT [10:16:59] yep [10:17:32] but before that it takes ages [10:17:36] Can someone fix that please ? [10:17:45] so just the one article? [10:18:11] can you edit the article as a whole? or does that also fail? [10:18:19] That fails too [10:18:22] ok [10:18:57] trying https://en.wikipedia.org/w/index.php?title=John_F._Kennedy&action=edit§ion=47 ... saving slow as well [10:19:24] but saved... [10:19:43] I cannot reproduce the error [10:20:31] huh [10:20:57] Nicolas_: can you try with another browser? (I'm trying to eliminate possible causes here one at a time) [10:21:33] Rollback works too [10:21:40] but amazingly slow [10:26:02] I just tried with another browser. Same problem. [10:26:08] also, is it slow to load pages or only slow to edit? (I assume you are logged in) [12:02:06] I am still uploading that 1.4 gb file since yesterday [12:02:20] I wonder if it is still continuing [13:35:24] *ahem* [13:35:48] * apergos gives emijrp_ the ol' hariy eyball :-P [13:35:51] *eyeball [13:36:19] evil has been done haha [13:36:20] * Nemo_bis hides new hard drive [13:36:38] emijrp_, it's getting full quickly without an uploader for the wikis ;) [13:37:23] i heard your are buliding a Internet Archive mirror at home. [13:37:47] god damn 20.5GB :P [14:07:13] Hydriz: 20.5GB? [14:07:25] hehe [14:07:46] some image dumps by Wikimedia :) [14:17:51] we are getting so so so so so so much bot operation at all of the wikis, the lack of security is a concern [14:18:32] the authentication seems to be so piss weak [14:42:05] Does Wiki Mail really hides senders information, like IP? [14:49:37] I have a problem with a page, it won't assess properly [14:50:58] Anyone online? [14:54:06] Specifically, http://en.wikipedia.org/wiki/Category:B-Class_United_States_articles_of_Unknown-importance [14:54:19] http://en.wikipedia.org/wiki/Talk:Choctaw [14:54:32] I've assessed the quality of this page yet it will not get out of the B-class of unknown importance category [14:58:12] Anyone? [15:39:03] Hi everyone. There's a Meta and mediawiki.org translation tools bug triage at 16:00 UTC in #mediawiki-i18n. That's in about 20 minutes. [16:16:33] Anyone on? [16:16:37] I need help with a broken page [16:20:10] ChrisGualtieri: seems fine to me [16:20:22] jdlrobson: ping [16:20:27] hey jeremyb [16:21:17] jdlrobson: you might want to fix https://otrs-wiki.wikimedia.org/w/index.php?title=List_of_accounts&diff=33371&oldid=33360 ;-) [16:21:58] err, https://otrs-wiki.wikimedia.org/w/index.php?diff=33380&oldid=33374 rather [16:23:04] Gyg? [16:23:05] you mean don't call myself AGUILAR Allan ? :) [16:23:07] Huh? [16:23:17] is that a default name? I don't remember ever editing this page [16:23:24] not a default [16:23:32] Its still at http://en.wikipedia.org/wiki/Category:B-Class_United_States_articles_of_Unknown-importance though [16:24:06] Though Choctaw is at a different catagory and it won't show up in the sorted catagory [16:24:35] jdlrobson: you has /msg [16:25:53] jeremyb: I think its an error because it doesn't reflect the change made, and I've refreshed it [16:27:33] ChrisGualtieri: refresh harder? i don't see what you're seeing [16:28:25] I just wiped out my cache again [16:28:45] how many articles do you see at that url above? [16:28:53] None or one? [16:30:53] @replag [16:30:56] jeremyb: No replag currently. See also "replag all". [16:30:58] @info enwiki [16:30:58] jeremyb: [enwiki: s1] db38: 10.0.6.48, db36: 10.0.6.46, db32: 10.0.6.42, db52: 10.0.6.62, db53: 10.0.6.63 [16:31:12] Its been like this since last night, several hours [16:31:32] !log aaron synchronized wmf-config/CommonSettings.php 'Removed backend config conditional now that everything was switched over.' [16:31:37] Logged the message, Master [16:34:34] jdlrobson: would you mind getting a cloak? [16:34:48] ChrisGualtieri: idk what's up there [16:34:55] i was looking at the wrong page [16:35:00] i see the same as you [16:35:08] (i was looking at the member page not the cat page) [16:35:09] When I edit the page I get an error [16:35:16] what error? [16:35:33] One sec I'll grab it [16:36:10] Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes. [16:36:23] jdlrobson: and /join #wikimedia-otrs #wikimedia-otrs-en [16:36:25] Request: POST http://en.wikipedia.org/w/index.php?title=Talk:Choctaw&action=submit, from 10.64.0.128 via cp1011.eqiad.wmnet (squid/2.7.STABLE9) to 10.2.1.1 (10.2.1.1) Error: ERR_ZERO_SIZE_OBJECT, errno [No Error] at Wed, 09 May 2012 07:06:49 GMT [16:36:47] ChrisGualtieri: apergos ping [16:36:49] gah [16:36:57] apergos: same as [[montreal]] earlier? [16:37:09] which seems like no one ever responded to you? [16:37:15] no one did [16:37:34] anyway, have to run [16:37:49] yeah, I'm mostly not here either [16:38:30] i hear you don't have a parliament yet. or whatever you call it [16:39:49] no government again today [16:39:53] its true [16:41:10] we should introduce legislations that allows apergos to travel [16:41:18] legislation* [16:41:31] I have a feeling that error is going to take awhile to fix [16:42:08] ChrisGualtieri: can you edit other pages? [16:42:17] Yes without problems [16:42:19] earlier when I looked through the various apache and syslogs, the only thing useful I saw were a pile of (newly generated) complaints about symlinks to skins-1.19 [16:42:29] nothing else remotely helpful [16:42:37] (for whoever else peeks at this) [16:43:24] i just got an unfancy (black and white) "Error reading from remote server" [16:43:44] didn't take long either [16:43:49] again [16:43:52] yuck [16:44:56] apergos: when i edit w/ no summary it takes a few secs and then it comes back with, you should use a summary (i have it enabled). when i save again then it takes a few more secs and comes back with that above [16:45:20] if you edit with a summary it works? [16:45:27] cause that sounds like a mw bug in that case [16:46:58] edit did go through actually: https://en.wikipedia.org/wiki/?title=Talk:Choctaw&diff=491627753&oldid=491543477 [16:47:45] w/ summary same error but edit did go through [16:47:48] (same page) [16:48:56] and the cat's still wrong [16:49:07] but that's a different issue i assume [16:51:10] then i edited choctaw again and it gave no error. and then talk:choctaw again and it did give the same error again [16:51:17] Well that's my issue actually [16:51:33] It breaks, won't update on the rest of Wiki, but the page catagory reflects the change [16:51:56] Rest of wiki doesn't put it where it belongs. [16:52:16] Not sure if it is a malformed WP:USA box or something triggering it with a template issue [16:52:44] But I tried updating it as it reflects to the other wikiprojects in hope of fixing it. Other pages sometimes have that problem, didn't try anything else. [16:54:08] !log aaron synchronized wmf-config/CommonSettings.php 'Make sure Swift backend will have journaling too.' [16:54:11] Logged the message, Master [17:23:06] robla: ping? [17:23:15] what's uP? [17:23:27] aren't we supposed to have a meeting right now? [17:23:37] or ~8 minutes ago :) [17:26:17] gah..sorry, yes [17:26:48] don't worry, free pizza's here [18:03:41] Reedy: where should we start deploying? [18:03:46] Dunno [18:03:53] Did you do it in stages before [18:04:20] 299 wikis to switch [18:04:27] yeah, with big wikis like this, it's probably good to do at least a couple individually, and then do the rest in one swoop [18:04:55] let's do dewiki first [18:05:20] de, fr, ja, ru are hte next big 4 [18:05:50] that works....let's do those in 5 min increments, and then the rest [18:06:03] de and fr should have users about [18:06:05] ru might... [18:06:09] I suspect ja won't [18:06:26] Let's start with DE then [18:07:49] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.20wmf2 [18:07:52] Logged the message, Master [18:09:04] I notified in #wikimedia-de [18:09:30] * jeremyb was just thinking that de was already done. but i was confusing with the sharded containers [18:10:03] lol [18:10:19] Reedy: alrighty. fr now? [18:12:08] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: frwiki to 1.20wmf2 [18:12:11] Logged the message, Master [18:14:48] Go for ru next [18:14:52] Reedy: yup [18:16:03] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: ruwiki to 1.20wmf2 [18:16:06] Logged the message, Master [18:20:34] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki to 1.20wmf2 [18:20:38] Logged the message, Master [18:23:47] * AaronSchulz watches Reedy blaze through the list [18:24:03] I'm going to migrate ALL of the wikis next [18:24:21] Reedy: yeah, I even made a test edit on ja....all looks good there [18:24:31] <^demon> Can we put MediaWiki.org back on 1.16 plz? [18:24:32] I say do everything else now [18:24:43] ^demon: can that wait 15 min or so? [18:24:51] <^demon> I guess :( [18:25:20] we should resolve the question of whether 1.15 or 1.16 would be better before rushing off [18:25:33] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 295 other wikipedias over to 1.20wmf2 [18:25:38] Logged the message, Master [18:25:49] * robla starts hitting all of the home pages [18:27:09] error log is still quiet [18:27:10] why limit to those 2 versions? [18:27:22] 1.6 would be better [18:27:24] maybe just have a different version per wiki? [18:27:30] pfft [18:27:32] <^demon> robla: After that, I've got a bug abotu running MediaWiki on Window 95 I'd like to discuss ;-) [18:27:39] 100k wikis look good. on t the 10k wikis: http://www.wikipedia.org/ [18:27:39] random wiki version per request [18:28:02] Reedy: ewww, that would be too evil [18:30:11] 10k wikis look fine [18:32:37] for a short while I wonder "what wikis is he talking about, we don't have so many" [18:32:59] * Nemo_bis archiving 7000 wikis [18:34:04] 1 PHP Warning: call_user_func_array() expects parameter 1 to be a valid callback, class 'languages' does not have a method 'getMessage' in /usr/local/apache/comm [18:34:04] on-local/php-1.20wmf2/includes/StubObject.php on line 59 [18:34:04] 1 PHP Warning: call_user_func_array() expects parameter 1 to be a valid callback, class 'languages' does not have a method 'getCode' in /usr/local/apache/common- [18:34:04] local/php-1.20wmf2/includes/StubObject.php on line 59 [18:34:04] 1 PHP Fatal error: Call to undefined method languages::getCode() in /usr/local/apache/common-local/php-1.20wmf2/includes/cache/MessageCache.php on line 616 [18:35:07] brion: can you cr https://gerrit.wikimedia.org/r/#/c/7023/ ? [18:35:30] * AaronSchulz debugs UW slowness [18:36:53] * Reedy waits to see if those errors are just transient [18:38:12] \o/ [18:42:28] robla: no change in what's appearing in the error logs [18:42:48] still lots of fatals? [18:44:57] "Cluster sql-s6: WARN - MNT-1228 Corrupted ruwiki." <-- Just got this error from a TS tool. Might be problematic? [18:47:14] The error message isn't useful [18:54:39] !log aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed fa1a8d5119e1174f7458eb9516287f4867c46484' [18:54:42] Logged the message, Master [18:57:04] binasher: I wish the stats sampling could be a bit more than 1/50 sometimes [18:58:49] site_stats is so terrible [18:59:06] Krenair: that's the TS, not relevant here. (you should probably read MNT-1228 if you want to know more about it) [18:59:08] we are in agreement :) [18:59:14] so so so bad [18:59:19] binasher: all it takes is one bad caller [18:59:23] and all others get blocked up [18:59:40] everything about it is bad [19:00:07] the db schema is terrible [19:00:08] binasher: it's like a stadium of people standing in line to write their name on a list [19:00:29] if you want to do counters in mysql, it should just be one row per stat, primary key of stat name and a value column [19:01:03] and nothing with high volume potential should be updated synchronously in a request [19:01:12] binasher: even then it might help to insert into some journal that is polled and updates the single value columns [19:01:32] ideally, job queue, but thats a mysql insert in the same db, lol [19:01:46] synchronously update a memcached counter [19:01:56] using compare and swap? [19:02:05] * AaronSchulz hugs the pecl memcache client [19:02:07] have a scheduled job that reads and sets the mc counters to 0 [19:02:16] update once per minute or something like that [19:02:38] i'm tempted to rip this apart as a live hack [19:03:01] if you needed multiple job procs, cas would be nice perhaps [19:03:03] * AaronSchulz wonders [19:03:07] anyway, lots of options [19:03:13] I'm amused there's php-memcached and php-memcache [19:03:36] binasher: Hmm I guess you're right that having separate stats in separate rows would improve things ^^ [19:03:40] I think memcached is the better one [19:03:42] * AaronSchulz can't recall [19:03:51] Not sure why anyone ever thought it was a good idea to put them in named columns [19:03:59] RoanKattouw: not enough, but a decent and obvious start [19:04:01] binasher: why not just use udp for the journal? idk what's being counted though [19:04:14] Hah, OOM in SqlBagOStuff.php [19:04:21] jeremyb: memcached does counters very efficiently [19:04:39] binasher: ohh, duh, atomic counter requests [19:04:41] binasher: idk how it works... is atomic? [19:04:42] i kinda want to look up who originally wrote SiteStats [19:04:46] AaronSchulz: yup [19:04:52] binasher: and what happens when you lose a node? [19:04:55] don't care? [19:05:07] we use them for "failed password" attempts and such counters [19:05:24] if the db is updated ever X seconds, i think we can afford to lose that amount of stats updates [19:05:39] way way better than having edits fail on lock timeouts [19:05:40] ok [19:05:55] https://graphite.wikimedia.org/render?from=-1weeks&until=now&width=800&height=380&target=query-m.UPDATE.site_stats_SET_ss_total_edits_ss_total_edits_N.tp99&uniq=0.03382439159984463&title=query-m.UPDATE.site_stats_SET_ss_total_edits_ss_total_edits_N.tp99 [19:06:01] that makes me rage [19:06:03] binasher: we can always have a monthly cron that fixes stats or such [19:06:21] binasher: puh, file uploads are getting blocking for 70s :) [19:06:27] sometimes 50 seconds of an edit save attempt is waiting for the lock on site_stats [19:06:29] sometimes, not always, thank goodness [19:06:31] wow.. [19:06:49] this is all.. [19:07:16] that's only 99%? sometimes it's worse then, right? [19:08:11] Author: Tim Starling Date: Tue Nov 21 09:53:45 2006 +0000 [19:08:17] binasher: 99% of dberrors.log was lock waits on delete for site stats [19:08:28] * AaronSchulz will keep an eye on that [19:08:30] yeah [19:08:32] well [19:08:51] we should redo this all right now [19:09:02] *right now*? [19:09:03] or turn it off [19:09:09] and I mean 99% as in 99/100 times, not percentile [19:09:12] did ya'll see the issues saving on [[montreal]] and [[talk:choctaw]] mentioned here earlier? [19:09:14] 99/100 errors I mean [19:09:49] its around 20% of the current log [19:10:18] yeah it fluctuates of course [19:10:28] plus I just committed some stuff ;) [19:10:39] Wed May 9 19:04:57 UTC 2012 srv258 commonswiki LocalFile::recordUpload2 10.0.6.41 1205 Lock wait timeout exceeded; try restarting transaction (10.0.6.41) UPDATE `site_stats` SET ss_images = ss_images+1 [19:10:55] hopefully those went down at least, hard to tell yet [19:11:22] binasher: complain to robla ;) [19:11:37] binasher: can we put everything in memcached? [19:11:39] it'll be quicker [19:11:56] * robla bumps around under the bus [19:11:57] Reedy: even files? [19:12:07] I said everything [19:12:14] robla: careful! [19:12:19] even the job queue :p [19:12:22] We can store our PHP code files in memcached too [19:12:36] also git [19:12:40] gerrit's db [19:12:45] woo [19:12:47] Have you guys looked at beanstalkd for a memcached-like queue? [19:12:50] Is memcached webscale? [19:13:13] me dropping tables is webscale [19:14:01] Domas might get jealous [19:14:52] i wish he was on right now [19:15:33] binasher: are you turning into domas? [19:16:20] I think I'm turning Lithuanian [19:16:20] I really think so [19:16:20] I'm turning Lithuanian [19:16:22] binasher: you want to get domas started? [19:16:24] i do wish i was a little bit taller [19:16:38] y'all [19:16:38] and heavier [19:18:59] AaronSchulz: that's easier ;) [19:20:45] Reedy: did you reforming CommonSettings? [19:20:49] *reformat [19:21:24] AaronSchulz: no... [19:21:34] weird, $wgFileBackends suddenly looks like ass [19:23:21] looks ok vim, so it must be ok [19:23:30] just the public noc one is ugly [19:26:25] hello domas – we missed you [19:27:00] binasher, robla: so we can work on a "quicker" fix of breaking site stats into multiple rows or work on something more long term [19:27:16] the current way looks pretty unsustainable [19:27:29] i just started on a bugzilla ticket fwiw [19:27:51] seemed more reasonable than 'drop table' :/ [19:28:13] heh....I guess you're not turning into domas :) [19:28:28] joking about dropping tables doesn't count [19:29:33] i thought i was getting domas level awesome but i can only dream [19:29:54] also you can be fired...which might be a factor [19:30:06] ;) [19:35:53] =) [19:37:50] you're not domas cool till you break the site a few times (and whilst on a plane iirc) [19:38:13] I fix sites while I'm on a plane [19:38:29] so it evens out [19:45:04] one way not to have contention on site stats is not to have insane amount of transactions hanging [20:37:22] CREATE TABLE /*_*/site_stats ( [20:37:24] -- The single row should contain 1 here. [20:37:25] ss_row_id int unsigned NOT NULL, [20:37:27] binasher: :) [20:38:29] AaronSchulz: are you rewriting it? :) [20:38:30] * AaronSchulz wonders what a good name for a new table would be [20:38:41] it can still be site_stats [20:39:09] wikistats seems fitting [20:39:11] I tend to prefer new names if totally redoing a table [20:40:30] binasher: first thing is to change direct write queries to go through a class [20:40:40] binasher: Do you see any issues with bumping the memory limit up a bit for MW processes? [20:40:52] Just wondering if it'll help with some of the OOMs... Or they'll just fail later [20:41:24] even moar?? [20:41:37] All of the memorys. [20:41:44] default is currently 120MB [20:41:54] zhwikis get 160 [20:41:58] unless it uses LC [20:42:01] err, 180 [20:42:05] yeah, 180 [20:42:16] how much do you want to bump it to? [20:42:22] Not sure [20:42:32] Certainly doubling isn't a good idea [20:42:36] woosters: have the 16 memcached hosts for tampa been ordered? [20:43:08] i'd like to have memcached move off the apaches before it's upped much [20:43:31] Ah [20:43:33] That's a good idea [20:43:43] I might just push it to 128 so it's base 2 ;) [20:43:49] slight bump now should be ok [20:44:01] binasher: the quotes for new mc machines in tampa, plus additional ones and memory upgrades for eqiad are with mark and ct for review ;] [20:44:11] so soon! [20:44:12] We get some silly OOMS like "tried to allocate 17 bytes" [20:44:38] those might still oom with an extra 8mb [20:44:56] yeah, hence me saying "or they'll just fail later" [20:44:56] !log reedy synchronized wmf-config/InitialiseSettings.php 'bump memory limit to 128MB' [20:45:00] Logged the message, Master [20:47:25] we should check how it changes the rate of ooms over the next day or two [20:48:14] Is that on gdash by any chance? [20:49:33] no, that's a good idea though. currently not injesting anything from the apache error / php exception logs [20:54:22] Allowed memory size of 134217728 bytes exhausted (tried to allocate 16777216 bytes) [20:54:45] Some of those really aren't going to get better ;) [20:57:36] binasher: do you want an rt ticket or something logging as a reminder to setup some stats for that? [20:57:51] Reedy: that would be great [20:58:11] I'll do that now. Thanks [21:11:24] Reedy is it normal for an upload to take more than a day? [21:11:26] file is 1.4gbs [21:11:27] No idea [21:11:55] Depends on your upload speed [21:11:55] I have a 10mb line [21:11:55] I dont have a progress bar :/ [21:11:56] look at a network monitor and see how fast it's transferring?> [21:12:32] binasher - waiting for mark to review [21:13:33] Reedy cant tell [21:13:33] I have a gigabit lan [21:13:51] 10mbs/sec doesnt pose much of a spike [21:15:04] bah canceled it [21:53:19] !log aaron synchronized php-1.20wmf2/includes/SiteStats.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227' [21:53:22] Logged the message, Master [21:53:50] !log aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed b9ac85cbf304a65d900cda00fafe53bf82d7a227' [21:53:53] Logged the message, Master [22:02:32] gn8 folks [22:08:54] Reedy or someone else: could we please put on the SOON TO DO list the update of Interwiki_map. Thx [22:09:04] Yeah [22:09:07] I'll do it in a few mins [22:09:18] I thought scap ran it too.. [22:10:03] when I checked the log yesterday, the last I saw was your from for 22 Mar [22:10:08] your run [22:10:18] binasher: I wonder if MEMORY tables would be useful [22:10:31] no [22:10:32] * sDrewth tries to get braining awake [22:10:36] not replication safe [22:10:38] * AaronSchulz recalls the hit_counter does something with those [22:11:05] What log? [22:11:09] binasher: you mean explodes with slaves or just doesn't replicate? [22:11:24] It's not speerately lodged [22:11:54] Hmm. Seemingly not [22:12:16] "The safe way to restart a slave that is replicating MEMORY tables is to first drop or delete all rows from the MEMORY tables on the master and wait until those changes have replicated to the slave. Then it is safe to restart the slave. " [22:12:28] https://wikitech.wikimedia.org/view/Server_admin_log (where you pointed me to check last time) [22:13:04] Yeah [22:13:12] But that's only logged if someone manually says so [22:13:31] I thought scap did it, semeingly not [22:13:33] [22:13:36] there are a few other gotchas, and a master crashing results in the slaves being wiped when it comes up, if its replicated, so provides no greater reliability than memcached counters [22:13:54] <- has no other means to know how else to check [22:14:08] binasher: not for reliability, for batch inserts [22:14:15] to use as a journal [22:14:16] Unless someone logs it, you don't really know [22:14:24] but why? [22:14:40] binasher: how would you use memcached? Just have a key for each stat? [22:14:45] and update using cas? [22:15:22] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [22:15:25] Logged the message, Master [22:15:53] well I hope that everyone then has a good systematic approach to logging, otherwise prodding, nagging and whining is the only tool available [22:15:54] each key being a "pending delta" accumulator [22:16:03] thanks reedy [22:17:00] [client 10.64.0.129] script '/usr/local/apache/common/docroot/wiktionary.org/w/index.php.PHP' not found or unable to stat [22:17:02] That's a strange one [22:17:49] cp1007 [22:18:40] binasher: in the long run, I'd prefer a journal approach [22:20:44] AaronSchulz: i'd key on wiki + stat. update with increment(), if fails due to key not existing, add() key with a value of one (that call fails if the key was created by another client in between those calls), if fails increment() [22:21:41] or we could introduce a persistent data store that does counters really well [22:21:50] i.e. redis [22:22:16] oh, yeah, I keep forgetting about increment/decrement [22:22:57] journals are good, but need to go somewhere where writes are really cheap [22:23:04] binasher: we could have either daemon handle the pending delta keys or an mt_rand() for web requests ala job queue [22:23:36] binasher: the way you proposed makes sense, since we are just doing ++ or -- [22:23:41] an mq type system could work instead of a db/file journal and still provide batching [22:24:16] but might as well use what we've got for now [22:26:02] we need a reliable distributed job scheduler too [22:32:30] !log Debugging ExtensionDistributor being broken. Likely to show more debug output on mw.org if you attempt to use it (though, it wouldn't give you what you wanted anyway) [22:32:34] Logged the message, Master [22:37:17] binasher: now I need to figure how to grap the pending updates and apply them without races [22:38:04] hmm, wouldn't be any issue with a single daemon though [22:38:56] meh, SELECT FOR UPDATE will work [22:42:29] yup [22:42:47] that makes sense to do actually [22:43:00] vs. the value = value + new value construct [22:48:38] RoanKattouw: can you apply https://gerrit.wikimedia.org/r/#/c/7122/ [22:48:49] * AaronSchulz just realized he is making changes in the wrong branch [22:48:51] sigh [22:49:22] Yeah you just merged that into mater [22:49:22] !log ExtensionDistributor fixed [22:49:24] *master [22:49:25] Logged the message, Master [22:52:07] RoanKattouw: no I was taking about some local changes, heh [22:52:14] Oh OK [22:52:17] Not too late then [22:52:29] AaronSchulz: Do you need me to do anything with 7122 still? [22:59:31] if you can deploy that would be sweet :) [23:01:57] OK [23:25:22] !log Created Translate tables on outreachwiki [23:25:25] Logged the message, Master [23:26:31] !log reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on outreachwiki' [23:26:34] Logged the message, Master