[03:38:25] !log on pc1: truncated tables pc158, pc159, pc160 due to flood of "table full" errors in dberror.log indicating innodb space exhaustion [03:38:37] Logged the message, Master [03:41:46] !log seems to have worked, also truncating pc161-174 to be on the safe side [03:42:00] Logged the message, Master [03:54:32] AaronSchulz http://ipv6test.wmflabs.org/wiki/index.php?title=Special:RecentChanges&days=30&from=&limit=500 needs to be cleaned up, but I can't login due to cookie errors... which I shouldn't be getting [03:56:55] TimStarling: are you able to clean that up using shell access? [03:59:07] uhhh [03:59:13] Jasper_Deng: that's labs? [03:59:18] yes [03:59:26] most labs wikis are spammed to death [03:59:26] so, why does it need tim? [03:59:34] I don't think it's really worthwhile to clean them up edit by edit [03:59:41] something's wrong with the cookies [03:59:43] I'd clean it up [03:59:46] but I can't login [03:59:46] I can delete the instance if you like [04:00:04] well, anonymous editing and account creation should get disabled [04:00:09] at the very least [04:00:12] yeah, thedj said that about the scribunto wiki [04:00:12] * jeremyb is starting to look... [04:00:18] until the cookie problem is resolved [04:00:38] he said, just give me access, I'll fix the 1000 spam edits per day, no problem [04:01:32] at least CheckUser is easy [04:02:44] checkuser? [04:02:51] on that wiki, all IPs are fake [04:03:00] so anyone can get CU [04:03:09] huh [04:03:22] I have CU access on that wiki [04:03:26] (except that I can't login) [04:10:43] !log restarted swift frontends [04:10:56] Logged the message, Master [04:15:55] Jasper_Deng: do you know what project that is? [04:16:03] idk [04:16:17] (Petrb is a likely founder) [04:16:22] * jeremyb can't tell... even querying LDAP directly [04:16:41] must be something i'm missing [04:17:09] anyway, brb, midnight snack [13:10:03] oh http://googlepolicyeurope.blogspot.it/2012/10/googles-data-centres-inside-look.html [13:10:59] space age blue lighting obligatory [13:12:01] http://www.wired.com/wiredenterprise/2012/10/ff-inside-google-data-center/ keeps the colour stile of the original http://infolab.stanford.edu/pub/voy/museum/pictures/display/0-4-Google.htm though [13:12:25] (tubes vs lego) [14:46:00] ^demon I've a little issue in Gerrit [14:46:06] it seems one repo is still broken [14:46:07] /home/dereckson/dev/mediawiki/mediawiki/tools/code-utils ] git-review -s [14:46:10] No '.gitreview' file found in this repository. [14:46:28] I did a git fetch --all and I've only an origin, no gerrit branch [14:46:28] Add one? [14:46:50] https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/tools/code-utils.git;a=tree [14:47:59] <^demon> It's not broken, it just needs a .gitreview file if you plan on using git-review. Like Reedy said, just add one. [14:48:35] Okay, done. [14:56:32] Dereckson: Did you commit the .gitreview file? [14:56:46] Yes, never mind [15:00:52] aye, ^demon have just merged it [18:00:42] Reedy: sumanah tells me there's a couple of things she's marked as blockers [18:00:49] I haven't had a moment to look but about to look now [18:01:02] I was just looking at 27441 as it would look like it's probably not in wmf2 [18:02:58] bug 41029 is features... for something on enwiki and nowhere else.. [18:05:49] Reedy: did you see the latest comment on that one? [18:06:17] which one? [18:06:39] 41029 [18:06:53] oliver says it has it's own window [18:06:53] Indeed [18:07:07] But our deployment would have no effect on it [18:07:07] jeremyb Reedy I'm not at all sure E2 has its own deploy window any more [18:07:21] As it's enabled on enwiki, and enwiki isn't due to be done till monday [18:08:05] i think it's generally pretty hard to get ironholds to stick in this channel. ;) [18:08:29] he is on though [18:10:15] Still [18:10:20] It's not relevant to the deployment [18:10:49] Reedy if you want that merged I'm pretty sure kaldari would do it, he's around in another channel somewhere [18:11:29] sorry, just discussed the parser cache situation with CT [18:12:01] We're not deploying to enwiki and nowhere else uses pagetriage (well, bar testwiki, but that's already upgraded).... [18:12:15] yeah, we can take that one off [18:12:28] (looks like that one was already marked off the list [18:12:46] so, the only one we're left with is the translatewiki issue [18:12:57] ...which, since we're deploying to meta, matters [18:13:12] we could just hold off on deploying to meta for that one [18:13:58] If indeed, its replicable on the cluster [18:14:18] well, here's the thing that really sucks about that one: [18:14:21] it could be a CPU eater [18:14:38] which, we're already in sketchyland with respect to Apache CPU [18:14:48] thanks to the parser cache issue [18:15:21] * chrismcmahon books travel to sketchyland [18:15:22] Aye [18:15:37] Seems to work for other translation pages, so it's not like all editing of translations is broken [18:17:09] I am skimming the list of recently filed bugs to see if there's anything else [18:17:12] more eyes are welcome [18:19:26] is https://bugzilla.wikimedia.org/show_bug.cgi?id=41033 important? [18:19:32] "Edits before page moves are shown in unexpected places in the watchlist" [18:19:36] Catchable fatal error: Object of class ParserOptions could not be converted to string in /usr/local/apache/common-local/php-1.21wmf2/includes/parser/Preprocessor_DOM.php on line 1088 [18:20:49] 2 more I'd like prioritization on: 41081: ExtensionDistributor does not work. 41078 - Special:ActiveUsers query is very slow. [18:21:10] Reedy: did you just deploy to meta? [18:22:07] yup, accidentally [18:22:08] easily reverted [18:22:23] <^demon> sumanah: 41033 sounds like a dupe. Pretty sure I've heard of that before. [18:23:05] activeusers is quick on enwiki [18:23:29] 41090 - upgradeRow() constantly triggered for invalid SVG and DjVu files [18:23:35] !b 41078 [18:23:36] https://bugzilla.wikimedia.org/show_bug.cgi?id=41078 [18:23:54] Reedy: bug 41078 is about master (as seen on twn) [18:25:00] Yeah [18:25:00] 41090 isn't new [18:25:35] <^demon> That's been like that for...a long time...forever? [18:25:51] <^demon> At the very least, since Brian's image metadata stuff. [18:27:45] sumanah: Very likely source for the special active users bug identified [18:28:07] https://gerrit.wikimedia.org/r/#/c/17945/ [18:28:25] ok, good to know [18:29:49] sumanah: Aren't you guys a bit late with announcing Wikimedia Bangalore DevCamp - Nov 9-11,2012? [18:30:02] multichill: you didn't see the save-the-date earlier? [18:30:14] there was an announcement several weeks ago, it just didn't have a reg link [18:30:28] Ah, just wondering :-) [18:30:46] 6157 unread mails in that folder so I might have missed a topic or two :P [18:31:33] robla: Reedy - so the current blockers to deployment are (I presume) possibly stuff with the parser cache, and Bug 41094 - Maximum execution time exceeded in MediaWikiMessageChecker.php ? If you're still working on that, I can do a little more poking to see if there's anyting else to catch [18:39:55] sumanah: I think maybe we just delay the deployment to meta, and ask someone from i18n to look at that problem tomorrow, or give us the all clear to deploy anyway [18:43:08] robla / Reedy / AaronSchulz: i have a javascript-only bugfix that i'd like to sync to production, but it's your window. do you think i could squeeze it in, or would it complicate things for you? [18:43:21] go ahead [18:43:23] much obliged [18:43:52] ori-l: which patch is it? [18:45:09] https://gerrit.wikimedia.org/r/#/c/28392/, which would roll out this https://gerrit.wikimedia.org/r/#/c/28289/ [18:45:29] ^ robla [18:46:56] Tool question, or should I ask on -toolserver? Which tool lets me see a distribution of file sizes AND image dimensions for non-free images (hosted at WP, not Commons)? [18:47:46] https://nv.wikipedia.org/wiki/Íiyisíí_Naaltsoos oh god my eyes [18:48:04] robla: is that ok? [18:48:04] * sumanah chooses not to look until she's told the nature of the eye-hurting [18:48:25] sumanah: color scheme [18:48:32] ori-l: that's fine [18:48:42] just wanted to double check what else was hitching a ride :) [18:49:02] (updating to master for that extension is going to pull more than just that one change [18:49:17] there is one php change, but it looks harmless enough [18:49:31] MatmaRex: Thx! Didn't know there was a Navajo WP [18:49:31] we're a little skittish right now because of load issues on the Apaches [18:49:46] robla: i was going to just sync-file, actually [18:50:04] okee doke, even better [18:50:13] cool, thanks again [18:51:01] MatmaRex: lol. [18:53:28] ori-l: Please don't say you are putting that on en-wiki! [18:54:19] Nathan2055: why not? [18:54:31] That color scheme. [18:54:42] It's scary! [18:54:50] oh, i have nothing to do with that :P [18:55:03] ( robla / Reedy: done, thanks ) [18:55:38] haha [19:13:15] hello, apparently this URL is a 404: https://commons.wikimedia.org/w/thumb.php?f=Broom%20icon.svg&w=22 [19:13:28] i think there is a similar bug going on right now, could youp oint me to it? [19:19:16] dawiki is running kind of slow - 28 seconds to load a diff... [19:21:04] yeah, the 'png' file is really just an html error from thumb.php [19:21:54] AaronSchulz: yes, i know this :) should i report it, or is it reported already? [19:22:12] is the same as https://bugzilla.wikimedia.org/show_bug.cgi?id=41113 ? [19:24:23] hello folks [19:24:35] I am getting a weird error while trying to see a deleted revision on Commons [19:24:38] "PHP fatal error in /usr/local/apache/common-local/php-1.21wmf2/includes/WikiPage.php line 3352: [19:24:41] Call to a member function getParserOutput() on a non-object" [19:29:28] odder: just logged that as https://bugzilla.wikimedia.org/show_bug.cgi?id=41127 with a full stack trace [19:29:57] thanks Reedy [19:30:27] Hmm [19:30:38] MatmaRex: Warning: Invalid argument supplied for foreach() in /usr/local/apache/common-local/php-1.21wmf2/skins/CologneBlue.php on line 631 [19:30:38] ;) [19:30:52] Reedy: uh, yeah, there's a patchset pending [19:30:58] but i think i needs to fix something in there [19:31:00] i need* [19:31:36] AaronSchulz: that could be it. would it help if i added that URL there? [19:31:51] Reedy: let's make a deal, i fix it now, and you merge it. :D [19:32:12] paravoid: https://commons.wikimedia.org/w/thumb.php?f=Broom%20icon.svg&w=22 [19:32:26] robla: I think I'm gonna revert the rest of the specials [19:32:32] so we have html 404 errors being stored as thumbnails, heh :) [19:32:33] We're getting 3 different new fatals [19:32:46] Reedy: open the floodgates! :p [19:32:50] Reedy: go for it [19:34:17] Reedy: the cologneblue thing is https://gerrit.wikimedia.org/r/#/c/27192/, i'm working on a PS right now. [19:35:48] Reedy: and there's Dantman's https://gerrit.wikimedia.org/r/#/c/27968/, which fixes the undefined index part of the issue :) [19:37:53] actually that wasn't a thumb.php error, it was some other 404 error [19:47:25] AaronSchulz: hahaha [19:47:49] I wonder what system made that 404 error [19:48:46] nice [19:51:15] reed [19:51:29] Reedy: i added you as a reviewer to the two cologneblue fixes. [19:51:33] Lucky me ;) [19:52:03] AaronSchulz: swift [19:52:13] Reedy: ;) [19:52:14] whatever it is, it's a localized error message [19:52:20] Reedy: you were just asking for it ;) [19:53:35] AaronSchulz: look at the body explanation = ('The resource could not be found.') [19:53:37] webob [19:53:53] so wtf is going on? [19:54:17] it's a thumb.php URL, you tell me :) [20:02:37] AaronSchulz: does api.php still have overhead that "regular" mw requests don't? [20:03:09] pretty much, minus some skin overhead [20:03:22] paravoid: ^^ [20:04:02] paravoid was just asking why we separate api requests to a different server pool [20:04:24] well the api sometimes gets flooded and goes down [20:04:31] that too [20:04:39] clicktracking and search debacles [20:17:35] That should've somewhat decrease in 1.21wmf2 with the removal of various reflection related type checks [20:17:41] domas said it took quite a lot of time [20:25:32] paravoid: odd, I deleted thumb/9/9e/Flag_of_Japan.svg/25px-Flag_of_Japan.svg.png via eval.php (which gave a failing status object), but now those urls work [20:26:35] er [20:26:48] the URL you gave me was not for the flag of japan [20:27:23] see https://bugzilla.wikimedia.org/show_bug.cgi?id=41113 [20:28:34] (NEW) Thumbnail generation infrastructure finally out of control - https://bugzilla.wikimedia.org/41130 blocker; Wikimedia: Media storage; (mr.heat) [20:28:36] lol. [20:28:48] haha [20:28:49] Reedy: has there been talk of reverting mediawiki.org to wmf1? [20:28:52] lol, out of control [20:29:04] robla: Not that I've seen [20:29:16] oh...wait, it looks like it already is [20:29:34] fscking dblists [20:30:13] the errors on mediawiki.org seem to hint that the problmes are related to the Translate extension [20:30:13] as are test and test2 [20:30:27] maybe disable that temporarily? [20:32:16] wth [20:32:54] Reedy: ? [20:33:06] Does mw.org look funky to you? [20:33:10] (performance is awful enough there that we should try do something there) [20:33:19] yes, it's messed up [20:33:26] GET https://www.mediawiki.org/wiki/.mw-translate-fuzzy%7Bbackground-color: 400 (Bad Request) load.php:90 [20:33:26] GET https://www.mediawiki.org/wiki/Li 404 (Not Found) [20:33:59] yep .. this does not appear with ?action=purge or ?debug=true [20:34:47] I've seen those happen before, I thought that was transient [20:34:55] hmm highlighting??? (linking just in case) [20:34:55] Wondering what the hell caused it; CSSJanus changes perhaps? [20:35:05] Reedy: Link to a page that's throwing those errors? [20:35:11] https://www.mediawiki.org/wiki/MediaWiki [20:35:13] after a purge I now seem to be able to load it fine [20:35:32] maybe parser cache corruption? [20:36:05] Maaaybe, but it's concerning [20:36:19] I suspect backwards incompatibility with Squid-cached pages [20:36:19] testwiki gets a load of 404/400 [20:36:20] or fear, rather [20:36:49] which testwiki page gets 404/400s for you reedy [20:36:57] main page [20:37:15] ailed to load resource: the server responded with a status of 400 (Bad Request) https://test.wikipedia.org/wiki/.mw-translate-fuzzy%7Bbackground-color: [20:37:15] Failed to load resource: the server responded with a status of 404 (Not Found) https://test.wikipedia.org/wiki/Li [20:37:23] Not seeing it [20:37:24] Failed to load resource: the server responded with a status of 400 (Bad Request) https://test.wikipedia.org/wiki/.ui-helper-hidden%7Bdisplay:none%7D.ui-help…mily:sans-serif;font-size:1em%7D.ui-widget-content%7Bborder:1px%20solid%20 [20:37:35] https://test.wikipedia.org/wiki/Main_Page works fine for me [20:37:43] yeah, can't repro it anymore either - do a clean cache view? [20:37:44] the urls it is trying to load seemt o have csss in them [20:38:02] Yeah [20:39:52] I manually pointed test.wikipedia.org to esams in my /etc/hosts and that still isn't hitting it for me [20:40:33] I am worried that a CSSJanus or RL change may be causing this, but I haven't been able to reproduce it myself so I can't investigate [20:40:49] I did see it happen on James's machine once, when testing VE on mw.org [20:41:21] I'm sure I just saw the php version being used change [20:42:08] "Tower X, Cybercity" O_o [20:42:23] oh that was for #mediawiki-i18n [20:43:17] Resource interpreted as Stylesheet but transferred with MIME type text/html: "https://test2.wikipedia.org/wiki/Special:Version". load.php:90 [20:44:13] The errors seem to have gone now... [20:46:27] I wonder [20:46:34] * Reedy tries to break it again [20:47:43] nope [20:47:44] MEH [21:22:28] does anybody know the timezone of tim? SF? [21:23:33] he's usually in australia [21:23:40] but maybe he's not there right now... [21:25:33] Well he's not here [21:25:38] I'd assume he's in Australia [21:25:53] (Sydney timezone) [21:26:01] Assumption is the mother of all screw ups [21:27:38] ok, so maybe online in a few hours (i only want to flood zhwiki job queue. i need him to run a script to clean doublicate jobs) [21:48:26] Does one exist? :/ [21:49:17] Merlissimo: there's already 1.4 million jobs.. [21:50:30] yes [21:50:44] wasn't below 1 miollion this week [21:51:12] tim wrote "but I could remove them all and then reinsert a single copy as a kind of duplicate removal" [21:51:56] after running my bot we maybe have 20-30 million jobs [21:52:24] Reedy: plop [21:52:47] Nikerabbit: ohai [21:52:49] I heard there were some issues related to Translate [21:52:53] Merlissimo: We can just truncate the whole table [21:53:28] hi robla [21:53:36] hey there [21:53:38] https://bugzilla.wikimedia.org/show_bug.cgi?id=41094 [21:54:45] Nikerabbit: Reedy: did you all figure out what's up with Translate and mediawiki.org? [21:54:53] (and meta, for that matter?) [21:55:52] (I've only got 5 minutes before next meeting, unfortunately) [21:55:59] Everything looking weird? [21:56:03] It fixed itself :/ [21:56:10] Reedy: i don't know which possibilities exist. on monday we decided that i'll run a script once on all country data templates, so that these template won't cause problem is future anymore after langlink updates. and tim wants to run a script when i am finished [21:56:24] nobody told me anything except 5 minutes ago via proxy [21:59:02] sorry...we were trying to debug. I don't remember if you were around to ping or not [21:59:33] (and I was only peripherally involved....I just asked Alolita since she was sitting right there) [21:59:59] that's not efficient [22:00:10] just ping me, I will answer if I'm around [22:04:43] Reedy: re 41094 indeed that code is not used on the cluster [22:05:40] heh [22:06:02] so it's certainly not any sort of a deployment blocker [22:08:44] Reedy: was there any other symptoms except broken css appearing in the wrong place? [22:09:28] The wiki was requesting strange urls, I'm not sure they were translate related at all [22:09:43] Reedy: examples? I can try to grep our access logs as well [22:09:57] GET https://www.mediawiki.org/wiki/.mw-translate-fuzzy%7Bbackground-color: 400 (Bad Request) load.php:90 [22:11:11] Generally URLs with /wiki/.mw- I guess [22:11:49] Reedy: Do you have referers for those? [22:11:49] Nope [22:12:17] nothing similar in our logs [22:14:39] Reedy: Did we deploy wmf2 more widely today? Did this issue surface at all outside of testwiki/mediawikiwiki? [22:15:01] I did to the specials, private etc [22:15:10] but there was 3 different fatals being pretty common, so I reverted it for now [22:28:59] Reedy: activeusers Q still slow [22:30:25] Nikerabbit: they're said to be broken in previous version [22:31:52] Nemo_bis: they=? [22:32:07] Nikerabbit: the queries [22:32:11] or maybe just oen [22:32:25] 101 rows in set (35.61 sec) [22:32:51] | 1 | SIMPLE | bw_recentchanges | index | rc_user_text | rc_user_text | 273 | NULL | 155136 | Using where | [22:32:54] | 1 | SIMPLE | bw_user | eq_ref | user_name | user_name | 257 | mediawiki.bw_recentchanges.rc_user_text | 1 | Using index | [22:32:57] | 1 | SIMPLE | bw_ipblocks | ref | ipb_user | ipb_user | 4 | mediawiki.bw_user.user_id | 1 | Using where; Not exists | [22:34:48] :/ [22:35:02] oh https://www.mediawiki.org/wiki/Special:Undelete/19h08min_de_19_de_Agosto_de_2011_%28hist%29_%28dif%29_Categoria:!Predefini%C3%A7%C3%B5es_para_o_dom%C3%ADnio_%22Anarcop%C3%A9dia%22_%28S%C3%B3_Voc%C3%AA%29 works again [22:35:38] I fixed something with it yesterday [22:37:42] Nikerabbit: any idea how long it's been slow? [22:38:13] The change before was essentially just syntax.. [22:38:35] Reedy: only noticed it in the profiling by accident few days ago [22:38:53] it could have been there for weeks as far as I'm concerned [22:48:18] hi mdale, have you seen that commons-l report? [22:56:30] Nemo_bis: thanks for highlight that [23:30:49] gn8 folks