[00:03:30] I'll merge https://gerrit.wikimedia.org/r/#/c/6894/ right now if there are no objections [00:04:31] <^demon> TimStarling: Reviewing. [00:05:53] I tested it locally [00:06:12] <^demon> Looks good, merged. [00:06:17] seems ok [00:07:01] I have to now merge it to two deployment branches, right? [00:07:42] and both require going through gerrit rather than a simple push? [00:07:42] <^demon> Cherry pick to 1.20wmf1 and wmf2, if we want it deployed right now. [00:07:57] <^demon> Yes, we disabled direct pushing for the deployment branches. [00:10:51] tricky [00:11:12] what's the easiest way to merge everything except the RELEASE-NOTES change? [00:11:27] <^demon> Reedy? [00:11:32] <^demon> How've you been doing it? [00:13:14] I think I just fixed the conflict [00:14:02] though, you could abandon the changes to that file [00:14:17] Well that's a valid way of "resolving" the conflict [00:14:41] I never bothered merging RELEASE-NOTES to wmf branches with subversion [00:15:24] makes sense [00:17:24] http://stackoverflow.com/questions/1526044/partly-cherry-picking-a-commit-with-git [00:17:34] cherry pick, reset, add, commit [00:17:54] right [00:17:56] thanks [00:20:47] !log aaron synchronized wmf-config/CommonSettings.php 'Switched enwiki to new backend config.' [00:20:50] Logged the message, Master [00:21:53] TimStarling: don't forget to check https://gerrit.wikimedia.org/r/#/c/6783/ when you get the chance [00:22:07] ok [00:31:03] the php- prefixes on our MediaWiki directories annoy me [00:31:20] you know that naming convention predates the name "MediaWiki", otherwise it would have been mw- or something [00:32:00] TimStarling: you can change it if you like, as long as nothing explodes :p [00:32:14] * AaronSchulz stirs the symlink soup [00:32:21] * AaronSchulz adds some salt [00:32:29] it's hard coded in multiversion now [00:32:40] not to mention scap etc. [00:32:48] and the apache configuration files [00:32:54] probably a lot of places [00:32:56] why the apache config? [00:32:57] You know there's a label on that soup pot saying "nitroglycerin 73% vol." right? [00:33:07] * AaronSchulz sighs [00:33:09] oh well [00:34:05] I thought that we would fix the prefixes when we did hetdeploy, it was in some notes I gave priyanka [00:34:13] but that idea was lost when the project transitioned to you [00:34:22] I thought I'd mention it in case another opportunity comes by [00:34:39] * AaronSchulz is a notebook now [00:36:47] !log tstarling synchronized php-1.20wmf1/includes/DefaultSettings.php [00:36:50] Logged the message, Master [00:37:12] !log tstarling synchronized php-1.20wmf1/includes/Exception.php [00:37:15] Logged the message, Master [00:42:10] !log tstarling synchronized php-1.20wmf2/includes/DefaultSettings.php [00:42:13] Logged the message, Master [00:42:32] !log tstarling synchronized php-1.20wmf2/includes/Exception.php [00:42:35] Logged the message, Master [00:43:56] exception.log is full of api.php spam [00:47:06] ah right, reedy broke it [00:47:10] typical [00:47:20] What's up with it? [00:48:10] // Log it [00:48:10] if ( $e instanceof MWException ) { [00:48:10] wfDebugLog( 'exception', $e->getLogMessage() ); [00:48:10] } [00:48:22] see, the instanceof was meant to make it not log UsageExceptions [00:49:37] Ah [00:50:39] I guess you're gonna revert it (no issue with that), would you add a comment to make it explicit that it's done for a reaosn? [00:50:43] it can probably be !($e instanceof UsageException) instead [00:50:51] that would make more sense, right? [00:51:17] That would, yeah [01:10:43] !log tstarling synchronized php-1.20wmf1/includes/api/ApiMain.php [01:11:07] !log tstarling synchronized php-1.20wmf2/includes/api/ApiMain.php [01:12:56] Logged the message, Master [01:12:59] Logged the message, Master [01:16:08] 2012-05-08 01:14:57 srv229 enwiki: [667e119d] /w/wiki.phtml?title=User:%C6var_Arnfj%F6r%F0_Bjarmason/monobook.css&action=raw&ctype=text/css Exception from line 914 of /usr/local/apache/common-local/php-1.20wmf2/includes/WebRequest.php: Invalid file extension found in the path info or query string. [01:17:12] I guess he used the .phtml extension just to be funny [01:17:36] it was .php long before action=raw was introduced [01:18:37] I wonder where that's being called.. [01:23:27] TimStarling: well how long has it been since any ^/w/wiki\. ? (should be /w/index.php , right?) [01:24:35] " using index.php and redirect.php instead of wiki.phtml and redirect.phtml [01:24:36] keeping phtml stubs around for compatibility and CVS history [01:24:36] " [01:24:53] Erik Moeller, Feb 2004 [01:25:28] lol [01:25:43] * RoanKattouw doesn't see Erik at his desk [01:26:28] he used to do a lot of development work back then [01:27:04] he was the one who suggested adding defaults to template arguments [01:27:10] i.e. {{{arg|default}}} [01:27:16] and we all know how that turned out, right? [01:27:37] haha [01:27:43] {{qif}} [01:27:46] yes [01:31:47] and action=raw was introduced in April 2004 by Gabriel [01:32:00] hah [01:32:02] so not much of a gap at all really [01:32:11] but I had the order right [02:00:16] !log LocalisationUpdate failed: git pull of extensions failed [02:00:20] Logged the message, Master [02:01:23] Server problems? [02:01:29] eww [02:01:31] * RoanKattouw looks [02:01:40] RoanKattouw: it did that like 4-5 days in a row now [02:01:51] error: Your local changes to '.gitmodules' would be overwritten by merge. Aborting. [02:01:53] Please, commit your changes or stash them before you can merge. [02:01:53] vssun1: like what? [02:01:54] * RoanKattouw rages [02:02:02] Local changes? On MY cluster? [02:02:25] sites are not loading / take long time to load [02:02:44] you could be a little more specific [02:03:08] getting better now.. I was looking for ml.wiktionary and ml.wikisource [02:03:36] git sucks [02:04:26] Reedy: be careful what you say, the british don't have seem to have the same freedoms of speech as US ;P [02:04:42] say what? [02:05:15] Reedy: "git sucks" [02:05:15] Chad isn't here [02:05:17] I'm safe ;) [02:06:22] vssun1: ya'll need to get a local logo made for mlwikt [02:07:03] otherwise i see no issues. (well if you discount me not having the right fonts) [02:07:36] might be some routing problems [02:07:54] now working fine. [02:08:11] sure, could be lots of things. could also be that i just didn't check until after it was half recovered [02:10:38] Thinking about the logo [02:10:44] will upload it soon [02:14:10] OK, l10nupdate should be fixed now [02:14:33] My workaround for the lack of magic submodules broke because we actually have magic submodules now :) [02:14:44] So I deleted and recreated the extensions checkout, should be all good now [02:17:37] does l10nupdate flush APC? [02:17:42] no [02:18:11] !log Removed and recloned /var/lib/l10nupdate/mediawiki/extensions , it was in a weird state because magic extension submodules work now but my hacky workaround for them not working was still in place [02:18:14] Logged the message, Mr. Obvious [02:18:15] jeremyb: Why would it need to? [02:18:52] idk, i figured it would have cached strings [02:19:11] No, APC doesn't do that [02:19:18] Not the l10n data at least [02:19:22] That's all cached in CDB & memcached [02:19:42] CDB? [02:20:01] .cdb files [02:20:11] It's a flat file key-value format [02:20:50] The l10n for each language is precomputed (because of fallbacks and because of l10nupdate overriding .i18n.php files in the MW source), then put in a .cdb file, then synced to the Apaches [02:21:02] So they just load a .cdb file into memory and they've got the entire l10n for a given language [02:21:55] oh, huh [02:22:02] Additionally, individual messages are cached in memc, that's populated from either .cdb (=disk) or the MediaWiki: namespace (=DB + external storage) [14:15:59] is there a list of wikimedia wikis that disabled local uploads? [14:16:51] http://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php Ctrl+F wgEnableUploads [14:27:35] heh [14:28:18] ToAruShiroiNeko, it's linked from the page I linked on your RfC: https://commons.wikimedia.org/wiki/Commons:Turning_off_local_uploads [14:33:35] anyone with access to http://dumps.wikimedia.org/android/ and http://dumps.wikimedia.org/iOS/ [14:33:38] ? [14:34:47] why? [14:35:02] It loads nicely for me [14:35:18] Hydriz: no, I am pushing out new betas, and want someone to upload files there for me :) [14:35:29] oh I see [14:35:38] ping apergos [14:35:48] * YuviPanda pings apergos  [14:35:51] he manages that server (I think) [14:36:13] I do [14:36:31] apergos: can you copy over two files there for me? [14:36:39] where are they now? [14:36:58] apergos: http://dl.dropbox.com/u/8768784/Wikipedia-v3.2-beta2.ipa (into /iOS) and http://dl.dropbox.com/u/8768784/Wikipedia-v1.2-beta2.apk (into /android) [14:37:33] apergos: At the same time update dvd.html to link to mlwiki/mlwikisource ISOs which I told you a few days ago [14:38:00] the dvd link is still on my todo [14:38:06] it take more than an edit in place [14:38:12] lol [14:38:45] ToAruShiroiNeko, I spammed your RfC in a few places [14:38:55] your files are there; please check that everything looks correct [14:39:24] apergos: thanks! checking [14:40:03] apergos: looks good. Thanks :) [14:40:07] sure [14:40:30] oh yes, does dumps.wikimedia.org cap bandwidth at 10MB/s? [14:41:57] for downloaders yes it does [14:42:13] I see [14:42:21] cos Meta was reporting 20MB/s [14:42:29] you can have two connections [14:42:43] zzz [14:42:52] anyways yeah we cap, sadly [14:43:08] which is why mirrors are a good thing [14:44:05] sigh [14:44:14] is your.org rsync working? [14:53:47] Nemo_bis thats ok [14:53:54] spam away :p [14:54:09] I want to avoid a flame war with the entire it.wikipedia if possible [14:54:15] I hope we don't have a flood of Commons users saying "noo we don't want more uploads to check please" [14:54:29] we might [14:54:36] the problem is mitigated to commons [14:54:38] but thats one place [14:55:07] if you promise to do the cleanup on Commons there will be no problems :) [14:55:22] I am not a commons admin so I doubt I can [14:55:47] the problem only lasts for one or two months then people give up [14:56:24] hm [14:58:45] I served as an admin when spanish wikipedia turned off their uploads [14:58:53] commons didnt die [14:59:11] btw how can I check if a value is a number or not with parse functions [15:53:41] if you revdel an edit summary on someone's userpage, that shouldn't trigger a notification email, right? [16:02:57] MBisanz, are you going to oversight Jimbo? :P [16:03:57] huh? [16:04:00] no [16:04:02] haha [16:36:42]  [17:00:20] Aaron|away: May 8 16:20:30 10.0.11.35 apache2[13592]: PHP Fatal error: require_once() [function.require]: Failed opening required '/home/wikipedia/common/multiversion/MWVersion.php' (include_path='.:/usr/share/php:/usr/local/apache/common/php') in /usr/local/apache/common-local/live-1.5/MWVersion.php on line 12 [17:00:36] Aaron|away: That's very weird cause 10.0.11.35 is not the testwiki host so it shouldn't be trying to access /home [17:00:46] [17:02:01] I pasted that one yesterday [17:16:56] !log reedy synchronized php-1.20wmf1/extensions/MobileFrontend/ 'Pushing out head' [17:17:00] Logged the message, Master [17:18:04] !log reedy synchronized php-1.20wmf2/extensions/MobileFrontend/ 'Pushing out head' [17:18:06] awjr: ^ [17:18:07] Logged the message, Master [17:18:10] Reedy: :O [17:18:21] Reedy: thanks! [17:18:34] Nikerabbit: yours is next [17:18:41] yay [17:19:26] mw35 code looks fine [17:19:59] I'd better fix the fixme on cr [17:20:02] * Reedy glares at AaronSchulz [17:20:23] ah, right [17:23:55] RoanKattouw, wrt https://bugzilla.wikimedia.org/show_bug.cgi?id=28026#c56 , how hard it is to change the current preference of all existing users from unset (NULL?) to false? so that we can then change the default to true [17:24:47] !log reedy synchronizing Wikimedia installation... : Rebuilding localisation cache files before TranslationNotification deploy [17:24:50] Logged the message, Master [17:25:14] Nemo_bis: I don't follow. There are two possible default values (true and false), not three (unset is not an option) [17:25:22] Oooh wait [17:25:25] I see what you're saying [17:25:33] sorry for bad terminology [17:25:36] Set all users' prefs to false explicitly [17:25:38] Right [17:25:39] yep [17:25:40] That could be done [17:25:55] You would need to set the default to true /first/ , then start setting stuff back to false [17:26:02] Otherwise it'll just convert false back to unset [17:26:50] RoanKattouw, and how long does it take to convert all users to false after setting the default to true? [17:26:53] It could be done fairly easily, someone would have to write a (fairly trivial) maintenance script to set all prefs to false [17:26:58] but it would take a bit of time to run [17:27:09] I can't tell you offhand how long it would take [17:27:31] hmm so one should kill the email relay to avoid spamming accounts? :) [17:27:31] But given that there are 20 million users on enwiki, I think it could easily take multiple hours [17:27:36] yeah [17:27:43] Hmm yeah you're right this is a problem [17:27:46] Well -- [17:27:58] We could hack the code so that the preference is offered but not actually functional [17:28:08] Reedy: [17:28:12] ops [17:28:16] Then default it to true, then start setting it to false for each user [17:28:19] Nikerabbit: ? [17:28:19] RoanKattouw: can't you modify database directly? [17:28:30] Then only when that is done, remove the hack such that the pref does things [17:28:33] RoanKattouw, could you comment on the bug explaining how to proceed for best? [17:28:38] Reedy: scap stil running, right? [17:28:41] Yup [17:28:59] Nikerabbit: Yes, but 1) you need to do it in 500-row batches so as to not make replication explode and 2) you need to invalidate the User obj cache in memc for every user you touch [17:29:11] Nemo_bis: Sure [17:29:18] thanks [17:37:37] Reedy: still running? [17:38:02] yup [17:38:15] sync done. [17:38:21] Nikerabbit: ^ [17:38:23] ;D [17:38:41] ~13 minutes [17:38:55] Reedy: yay [17:39:01] Reedy: so now I can go and break the site? [17:39:05] Sure [17:39:08] Where are you enabling it? [17:39:42] Reedy: mworg and meta [17:39:49] and maybe incubator and wikimaniawiki too [17:39:53] Cool [17:40:47] is it technically possible to setup a local wiki that uploads to commons instead of locally? [17:40:57] provided it is a sul account [17:41:15] but doing so using the local form and not commons forms [17:41:32] Yes, it would be [17:42:10] was that a reply to me? [17:42:56] ToAruShiroiNeko, that's a feature under development AFAIK [17:43:27] AaronSchulz: has someone made you aware that InstantCommons is broken? https://translatewiki.net/wiki/File:Yes_check.svg / https://translatewiki.net/wiki/Thread:Support/Template:Rights_using_non-existing_image [17:43:54] <^demon> siebrand: I know we got some fixes in for IC a couple of days ago. [17:44:07] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on mediawikiwiki' [17:44:10] Logged the message, Master [17:44:55] Nemo_bis how much "under"? [17:44:56] I'd expect that tests would catch these kinds of breakages. How could this get broken in master HEAD without something firing? [17:45:36] ToAruShiroiNeko, currently only planned IIRC [17:45:47] <^demon> siebrand: There aren't any tests for InstantCommons. [17:45:54] ToAruShiroiNeko, just don't mention it anywhere or bikeshedding will kill your RfC ;) [17:46:05] simple links to Commons from the sidebar are perferctly ok [17:46:23] ^demon: okay. Will open an an issue in bugzilla. [17:47:00] <^demon> The problem with instant commons tests is they'd require external requests (which was part of the flaw with the HttpRequest tests). [17:47:17] <^demon> We need to make a new group like @externalhttp or something so we can exclude them by default. [17:47:45] Nemo_bis somepeople find the default forms confusing [17:48:00] ToAruShiroiNeko, who cares about default forms? [17:48:04] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable TranslationNotifications on meta, incubator and wikimania2012' [17:48:07] Logged the message, Master [17:50:31] thanks for the comment RoanKattouw :D [17:50:42] now I only need to know who could do that... :[ [17:53:53] ^demon: done: 36653/36655. [17:54:03] <^demon> gerrit's down. [17:54:11] <^demon> Investigating why. [17:54:20] <^demon> Networking issue [17:56:20] <^demon> siebrand: Back [18:00:36] RoanKattouw can i bug you with some git questions real quick? [18:00:59] Suer [18:01:29] RoanKattouw: i need to roll back MobileFrontend to a previous revision, and then cherry pick a change on top of it in deployment [18:01:40] but im not entirely sure how to do that [18:01:43] Hmm [18:01:58] Do you currently have a deployment branch for MF? [18:02:00] basically, i want to roll wmf1/wmf2 back to https://gerrit.wikimedia.org/r/#/c/6869/ [18:02:00] deploy from SVN XD [18:02:16] and then apply https://gerrit.wikimedia.org/r/#/c/6915/ [18:02:26] RoanKattouw: yes we currently have a deployment branch [18:02:38] Hmm OK [18:02:58] How many commits are you seeking to revert? [18:03:31] RoanKattouw: 6, i think, including the merges [18:04:27] OK [18:04:39] RoanKattouw: would it be easiest to blow away our wmf/1.20wmf1 branch, start over again by creating the branch from point in the past, then cherry pick the change on top of it? [18:04:45] So really what you should do is just revert those individually in the deployment branch [18:04:55] Because blowing away a branch is so rude that Gerrit prohibits it [18:04:59] lol [18:05:00] ok [18:05:44] 'git revert ab56f7....' will create a revert commit, and you'll need to run 'git commit --amend' after that to make sure it gets a Change-Id [18:06:01] You'll want to revert all commits that need undoing, skipping the merges [18:06:32] <^demon> RoanKattouw: It's only blocked because you can't force push and the syntax is awkward ;-) [18:07:07] ok in that case i'm just reverting three changes [18:07:18] OK cool, that's managable [18:07:24] manageable ? [18:07:31] should i amend after i revert all three? [18:07:37] or amend after each revert? [18:08:10] amend after each revert [18:08:14] Alternatively [18:08:27] No, wait, never mind, that won't work [18:08:28] ignore me :) [18:08:29] oops [18:08:41] Yeah just amend after every revert [18:08:51] I was going to suggest something else and then I realized that that wouldn't work [18:08:53] ok i jumped the gun and reverted them all first [18:08:57] with no amend [18:09:00] Oh OK [18:09:06] uh [18:09:06] So none of the revert summaries have change-IDs? [18:09:08] That can still be fixed [18:09:10] currently no [18:09:19] OK [18:09:29] There's 3 of these right? [18:09:35] correct [18:09:39] Then run git rebase --interactive HEAD~3 [18:09:45] oh yeah [18:09:46] makes sense [18:09:57] That'll throw you into an editor listing all your revert summaries and nothing else (hopefully) [18:10:04] correct [18:10:08] pick 'em all? [18:10:08] For each revert summary, change 'pick' to 'edit', then save and exit [18:10:10] k [18:10:12] Ah sorry [18:10:14] 'reword' [18:10:17] s/pick/reword [18:10:28] ok [18:10:42] and then save each one? [18:10:44] Then that'll allow you to edit the commit summary of each revert in order [18:10:48] And just save those without changes, yes [18:10:55] cool [18:10:55] now [18:10:57] git review? [18:11:01] Then look at 'git log' to check they all have Change-IDs [18:11:11] yes they do [18:11:14] Sweet [18:11:17] Then yeah git review [18:11:28] It'll warn you about submitting multiple changes but that's fine [18:11:35] Because, you know, you are [18:11:42] hehhehe ok cool thanks [18:11:49] then i'll do the same thing on the other branch [18:12:43] Nemo_bis should I mark supports and opposes with template? [18:12:56] oh woops [18:13:03] i should have cd'd out of my extension before git review i think [18:13:20] ToAruShiroiNeko, dunno, only if it advantages you of course :D [18:13:50] adding templates might be considered altering comments, but moving them to sections is acceptable refactoring (if it doesn't alter threaded replies) [18:15:04] awjr: No, you should git review from inside the extension too [18:15:15] o [18:15:25] It should have submitted 3 commits for MF and one for core[wmf/1.20wmfN] [18:15:36] shit ok i think i screwed this up [18:15:41] Nemo_bis I suppose so [18:15:47] I shall leave them as is then [18:15:56] RoanKattouw: it submitted three changes from inside the extension to master; i abandoned those. [18:16:00] Oh [18:16:06] You should un-abandon them [18:16:10] lol ok [18:16:12] Oooh it submitted them to *master* [18:16:13] OK [18:16:15] In *that* case [18:16:20] Keep them abandoned [18:16:36] But resubmit them (from inside the extension dir) with 'git review wmf/1.20wmf2' [18:16:39] ok yeah that's what i thought [18:18:16] aw crap [18:18:19] i think i need to rebase again? [18:20:42] RoanKattouw woot [18:20:58] I thought I was the master [18:21:25] awjr: ? [18:21:41] awjrichards@virtualbot:~/Dev/wikimedia_git/deployment/core/extensions/MobileFrontend$ $ git review wmf/1.20wmf2 [18:21:41] Errors running git rebase -i remotes/gerrit/wmf/1.20wmf2 [18:21:42] fatal: Needed a single revision [18:21:42] Invalid base [18:21:45] RoanKattouw ^ [18:21:46] awjr: Your commits were based off the wmf/1.20wmf2 branch I hope? [18:22:00] i believe so yes [18:22:03] OK [18:22:08] Then rerun it with -R [18:22:20] RoanKattouw if you have the time I'd like to show you a few bugs [18:22:56] ToAruShiroiNeko: Today's my 20% day anyway, so hit me [18:23:28] ok [18:23:42] https://bugzilla.wikimedia.org/show_bug.cgi?id=21338 [18:23:49] https://bugzilla.wikimedia.org/show_bug.cgi?id=22521 [18:23:49] gah RoanKattouw: [18:23:50] awjrichards@virtualbot:~/Dev/wikimedia_git/deployment/core/extensions/MobileFrontend$ $ git review wmf/1.20wmf2 -R [18:23:50] Had trouble running git log --decorate --oneline wmf/1.20wmf2 --not remotes/gerrit/wmf/1.20wmf2 [18:23:50] fatal: ambiguous argument 'wmf/1.20wmf2': unknown revision or path not in the working tree. [18:23:50] Use '--' to separate paths from revisions [18:23:56] https://bugzilla.wikimedia.org/show_bug.cgi?id=32077 [18:24:02] https://bugzilla.wikimedia.org/show_bug.cgi?id=34041 [18:25:04] RoanKattouw i think i really screwed this up i think i should just start over [18:26:30] awjr: wtf... do you even have a local wmf/1.20wmf2 branch? [18:26:40] RoanKattouw: i do, at least for core [18:26:45] And for the ext? [18:26:53] wtf [18:26:55] i thought i did [18:27:01] but i appear to be in a detached state [18:30:21] ok i see, MobileFrontend did not have a wmf/1.20wmf2 branch [18:36:14] RoanKattouw any remarks? :0 [18:36:44] ToAruShiroiNeko: Oh sorry lemme look at those [18:38:26] brion: So what's the state of chunked uploading now, do you know? Wasn't it like enabled experimentally last week? [18:40:51] RoanKattouw, i believe it should work; last i saw erik was requesting it be turned on but i forget if we did yet [18:40:54] lemme check the live config [18:41:09] brion: Where "it should work" means UW supports it as well? [18:41:23] $wgUploadWizardConfig['enableChunked'] = 'opt-in'; in CommonSettings.php [18:41:29] so yeah, UploadWizard should be using it where enabled [18:41:32] brion would ordinary users be able to use it? [18:41:44] ToAruShiroiNeko, if you write an API client to upload in chunks, then yes :) [18:41:50] or else use UPloadWizard [18:41:52] ah [18:41:53] thought so [18:42:01] I want to upload very large files [18:42:03] Special:Upload doesn't have any logic for it [18:42:07] I want to avoid coding for it :/ [18:42:13] An idea is pywikipedia implementaiton [18:42:37] pywikipedia could indeed implement chunked upload, if it hasn't yet [18:43:12] it probably hasnt [18:43:29] pywikipedia always lacks a feature when I needed, all others are available :/ Murphy! :p [18:43:32] the other part of the puzzle is making sure larger filesize limits get applied for chunked uploads [18:43:36] heh [18:43:39] brion: So how does one opt into UW using it? [18:43:51] RoanKattouw, preferences, under 'upload' i think [18:43:56] brion: Yeah we're talking abuot that (bug 21338) now [18:43:58] OK [18:44:16] on commons the general consensus is to allow very large files to only select users [18:44:22] 'Experimental features [18:44:22] Chunked uploads for files over 1MB in Upload Wizard' [18:44:23] such as a very large upload group [18:44:53] some troll's gonna upload a 200gig junk file :) [18:45:24] i'm actually not sure if/how the file size limits are currently applied to chunked uploads, should probably check [18:47:40] They should b [18:47:47] Or at least I just claimed they are in my BZ comment [18:48:36] brion commons would perhaps want a special user group that can upload large files and also be able to delete them permanently [18:48:50] is performance a concern if people accidentaly upload a 20gb file [18:49:04] ToAruShiroiNeko: I just commented https://bugzilla.wikimedia.org/21338 and suggested a 1GB limit [18:49:14] Currently FileBackend has a 4GB hard limit anyway [18:49:21] I already have a 1.2 gb video I would want to upload [18:49:33] Which in turn is because Swift can only do up to 5G [18:49:47] the inaguration of putin which has prety dazzling views of kremlin [18:49:52] Alright, so make the limit 4GB, I don't think that's a problem [18:49:54] <^demon> brion: Remember when our jpgs were like < 1MB? Those were the days... [18:50:20] ^demon its a letter difference [18:50:23] M->G [18:50:24] ^demon: s/jpgs were/ram was/ [18:50:25] big deal :p [18:51:02] ^demon, yeah we also didn't have any thumbnailing back in the day. kept things simple! [18:51:13] RoanKattouw: swift has support for bigger files. but maybe not the mediawiki glue for swift [18:51:54] I think AaronSchulz said >5GB was a problem for Swift, but I don't know wh [18:51:56] y [18:52:10] Anyway, the MW file backend currently imposes a hard 4GB limit [18:52:10] the reason why is to make the cluster more balanced [18:52:28] well that's an assumption [18:52:31] anyway [18:52:56] i don't remember the exact procedure for uploading bigger files but it's approximately: upload chunks that are small enough (4 or 5GB or whatever you like) [18:53:12] and then upload a manifest file which just points to a list of other files [18:53:40] when you download that "manifest" file it actually retrieves all of the parts cat'd together [18:53:54] transparently. 1 GET [18:55:09] right [18:56:55] ToAruShiroiNeko: Re bug 22521 , it's nontrivial, you would have to find a dev who's interested [18:57:30] mm [18:57:37] ToAruShiroiNeko: bug 32077 I don't really understand very well, can't really comment on that [18:57:41] I tried different baits, no dev has been biting :p [18:58:03] RoanKattouw basically mediawiki logs a user off if an unrelated account is renamed [18:58:43] re bug 34041 , complain to the person that broke it (looks like that was Tim) [18:58:56] so if [[User:Foo]] is renamed to [[User:RoanKattouw]], on ja.wikipedia, your en.wikipedia account would be logged off. [18:59:04] ToAruShiroiNeko: Yeah I understand why the behavior is bad, but I don't understand why it happens or how one would fix it [18:59:58] RoanKattouw my guess is it manipulates the cookie [19:00:22] It invalidates the cookie for all accounts with that username [19:00:26] Well there's both local and global-ish (CentralAuth) cookies, God knows what kind of interaction goes on there [19:00:55] I think it kills the global coookie which in turn kills all local cookies [19:02:23] RoanKattouw OR the cookie monster is set loose on the servers [19:02:47] heh [19:03:03] I am having difficulty convincing that this is a bug [19:03:45] A sul rename would fix it but thats as unlikely as manned mission to the sun [19:04:06] RoanKattouw: does the current system have any files > 5G? [19:04:34] I don't believe so [19:06:07] jeremyb data dumps maybe :p [19:06:24] uhuh [19:06:40] i think no one's uploading data dumps to commons [19:08:25] not yet [19:08:31] >:D [19:10:57] !log awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ [19:11:00] Logged the message, Master [19:11:22] !log awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ [19:11:25] Logged the message, Master [19:15:13] images > 5mb? [19:15:20] there are videos on there bigger than 1gb [19:15:24] jeremyb: [19:15:46] He was asking for >5GB [19:15:47] and I know for a fact there are straight up images > 10mb [19:15:52] hmm [19:15:53] Which I'm pretty sure we don't have [19:16:00] only if there's a video [19:16:27] toolserver query would answer that question [19:16:44] or wading through the dump of the image table :-P [19:17:45] max(img_size) = 3451828359 [19:17:59] So that's ~3.3GB [19:18:22] 2012_State_Of_The_Union_Address_(720p).ogv [19:18:26] bc says: scale=2;3451828359/1048576=3291.92 [19:18:37] obama talks too long :-P [19:18:44] * jeremyb has an alias for that ;-P [19:20:10] and we have odderbot to blame for it [19:20:26] but i guess it was not actually uploaded by odderbot [19:20:33] That was probably me or Reedy actually [19:21:13] how about wikimania videos? none bigger? [19:21:29] I guess they would all be on commons, nt on the separate wmania site eh? [19:22:20] yes, commons [19:23:11] I'm working on the WM11 videos now [19:23:57] good Roan [19:24:48] Largest that I've found is 2.4G Wikimania 2011 - 3rd day Chapters and co. II.ogv [19:25:51] hehe the usual endless chapters discussions [19:34:46] !log awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ [19:34:49] Logged the message, Master [19:38:23] RoanKattouw, I added https://meta.wikimedia.org/wiki/Wikimania_Handbook#Upload [19:40:24] Nemo_bis: Thanks [19:43:07] apergos: I've got a batch of Wikimania videos ready, 18 videos for a total of 17GB. Would you like me to import those in a staggered way or can I just import them all today? [19:46:30] they can all go today but can you do me a favor and [19:46:41] you'll kill me but it's so my replication doesn't fall too badly behind [19:46:46] I assume you have a script? [19:47:10] RoanKattouw: [19:50:02] apergos: I'd just use importImages.php [19:50:20] Do you need me to sleep for a certain amount of time after each file? [19:50:28] well I was thinking of it [19:50:39] if putting a 15 min wait in there wouldn't be a drag [19:50:41] you could copy first to the replica and *then* do the import? [19:50:43] remember you need to sudo -u apache [19:50:48] Yeah [19:50:57] Ahm that's yes @ Reedy [19:51:00] No @ jeremyb [19:51:04] hehe [19:51:09] and no we can't copy to the replica host, it's zfs [19:51:11] apergos: 15 min * 18 files = 4.5 hours, that's reasonable [19:51:16] with fancy snapshot replication [19:51:21] ok, go to town [19:51:22] thanks [19:53:21] can't wait for zfs to be gone [19:54:48] * jeremyb hands apergos a btrfs? [19:54:52] !log aaron synchronized wmf-config/CommonSettings.php 'Moved remaining wikis over to new backend config' [19:54:55] Logged the message, Master [19:55:02] maplebed: that was not the sharding btw [19:55:18] I wish. I keep looking at the status but it's too soon [19:55:18] heh... thanks. [19:55:36] i figured [19:56:17] i think swift is fine though. just through some good offline backups (maybe distributed too) on there as a last resort to restore from and you're good [19:56:30] mirror [19:56:43] for the images we'll have it, but it might be a few days behind [19:58:53] RoanKattouw how can I use chunked uploads? [19:59:33] ToAruShiroiNeko: Using UploadWizard, or using the API [19:59:43] For UW, you have to enable a preference [20:00:01] Note that chunked uploading doesn't currently allow uploading >100MB files [20:00:09] But the technology is a prerequisite for allowing that [20:00:13] i thought it did? [20:00:22] RoanKattouw? [20:00:30] 500M or something iirc [20:00:53] I don't /believe/ so [20:00:55] I could be wrong [20:01:03] RoanKattouw: erik said it did [20:01:07] OK [20:01:12] (because the check is circumveted in some way) [20:01:16] He's probably right then [20:04:20] AaronSchulz: would you review the change I made to http://wikitech.wikimedia.org/view/Swift/Deploy_Plan_-_Shard_Smaller_Wikis [20:04:38] Erik is right but chunked uploading is broken anyway [20:06:38] RoanKattouw why is it capped at 100mb? [20:06:56] I need to put in [20:07:04] I did enable the UW [20:07:23] the uW or the chunked uploading [20:08:10] maplebed: seems ok [20:08:40] AaronSchulz: ok, running... [20:08:45] * AaronSchulz isn't super familiar with using swift via curl [20:08:57] err... that's not via curl. [20:09:02] UW + chunked uploading [20:09:06] lol, right [20:09:10] via swift tools [20:09:52] Nemo_bis it involves https://gerrit.wikimedia.org/r/#/c/6722/ I believe [20:10:19] ToAruShiroiNeko, didn't upload start at all? [20:10:29] what OS and browser did you use? [20:11:54] windows ie [20:11:57] it starts but dies [20:12:00] file is over 100mb [20:12:10] its an ogv and has putin [20:13:15] again that Putin video?? [20:13:31] ToAruShiroiNeko, try Chrome [20:14:01] !log creating sharded containers for swift for 'dewiki','fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki' [20:14:04] Logged the message, Master [20:14:09] it's weird btw, on Ubuntu it seems to work better with Chromium and on Windows with Firefox [20:16:56] AaronSchulz: I'm only creating the thumbnail sharded containers, not sharded containers for originals etc. that's ok, right? [20:17:12] great [20:17:20] maplebed: yeah, just thumbs [20:17:24] that chrashes chrome incognito [20:17:59] nop [20:18:04] crashes regular chrome too [20:18:41] Nemo_bis can I panic now [20:18:53] no [20:18:56] try ff [20:19:07] and if it's not enough, dump windows [20:22:33] apergos: OK starting the first upload now [20:22:42] w/ 15min sleeps [20:23:04] cool [20:23:30] I"m not going to bother to check, replicaation might fall a little behind but it will have time to catch up [20:23:35] thanks [20:23:43] Thank you for accommodating [20:23:51] apergos: is it in ganglia? [20:24:37] no, it runs as a cron job and sends an email (to local root) when it runs pastits 15 minute time slot.. the next one wont' fire til thelast one's done is all [20:24:39] no biggie [20:24:46] but it would suck to get too far behind [20:25:14] Nemo_bis I can upload the video using meego if you like [20:25:26] or how about dos? [20:25:38] dos might work [20:25:50] try mac os 8 [20:30:56] !log reedy synchronized php-1.20wmf2/extensions/AntiBot/ [20:30:58] ToAruShiroiNeko, you're wrong, that's not the limit [20:31:00] Logged the message, Master [20:31:11] https://commons.wikimedia.org/wiki/Special:Contributions/prolineserver [20:31:19] maplebed: how goes the script? [20:31:28] !log reedy synchronized php-1.20wmf1/extensions/AntiBot/ [20:31:30] it's half way through id. [20:31:32] Logged the message, Master [20:31:51] so about 30% done. [20:32:28] it's creating them at a rate of about one container per second, [20:32:46] so should be done in 35 minutes or so. [20:32:54] hmm... [20:33:02] * AaronSchulz though that would be a bit faster for some reason [20:33:07] I'm going to start multiple copies [20:34:15] Nemo_bis the form puts the limit at 100mb [20:34:20] it was also the quoted limit here [20:34:55] AaronSchulz: drop that from 35 to 15 minutes. [20:35:06] (I just started a second copy) [20:49:26] maplebed: it would be funny if all the containers had a typo [20:49:36] funny ha ha. [20:49:59] btw, would you reviouw https://gerrit.wikimedia.org/r/#/c/6954/ [20:51:33] almost done. [20:53:41] AaronSchulz: all containers created successfully. [20:54:05] then I can make the change [20:54:26] !log Created translate related tables for wikimania2013wiki [20:54:29] Logged the message, Master [20:54:55] :D [20:55:06] AaronSchulz: go ahead. [20:56:21] !log aaron synchronized wmf-config/swift.php 'Switching purge hook to use new sharding scheme.' [20:56:24] Logged the message, Master [20:56:34] !log reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on wikimania2013wiki' [20:56:37] Logged the message, Master [20:56:40] hrm [20:57:02] AaronSchulz: hmm? [20:57:17] !log reedy synchronized wmf-config/InitialiseSettings.php 'remove w' [20:57:20] Logged the message, Master [20:57:21] should I not continue? [20:57:36] I think it was reedy's fault [20:57:52] Yeah, I'd got a w before looks good now [20:58:22] AaronSchulz: so I should push the change to rewrite config? [20:58:29] Reedy: preilly and I spend like 2 hours tracking one of those down in a patch last week [20:58:33] maplebed: sure [20:59:00] Reedy: it was a ' ' in that case :) [20:59:01] pushing to ms-fe1 [20:59:09] * AaronSchulz pulls the graphs [20:59:32] !log reedy synchronized wmf-config/InitialiseSettings.php 'logo for wikimania2013wiki' [20:59:35] Logged the message, Master [21:04:16] AaronSchulz: looks like it's working on ms-fe1 [21:05:02] I requseted http://upload.wikimedia.org/wikipedia/de/thumb/2/23/2eek.jpg/220px-2eek.jpg specifically from swift and it's now the only object in the de thumb,23 shard [21:05:55] there are also 11 other objects in the sharded containers just from regular traffic. [21:06:58] pushing it to ms-fe2 [21:07:19] Krinkle: dbbot-wm's having a bad day [21:07:43] erm, ok then! [21:10:11] jeremyb: thx [21:10:34] grr [21:10:39] I blame toolserver [21:10:43] my script hasn't changed in weeks [21:11:00] you mean you blame SGE? [21:11:06] @info db45 [21:11:06] jeremyb: [db45: s5] 10.0.6.55 [21:11:06] jeremyb: [db45: s5] 10.0.6.55 [21:11:07] jeremyb: [db45: s5] 10.0.6.55 [21:11:08] among other things, yes [21:11:29] !log reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki' [21:11:32] Logged the message, Master [21:11:53] it is starting stuff that is already active, even though cronsub (the way I use it), should not do that if an sge with the name is already active [21:12:25] !log reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki' [21:12:29] Logged the message, Master [21:13:42] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move wikimania2013wiki to php-1.20wmf2 [21:13:44] !log delpoyed container sharding for thumbnails to swift for 'dewiki', 'fiwiki', 'frwiki', 'hewiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'rowiki', 'ruwiki', 'thwiki', 'trwiki', 'ukwiki', 'zhwiki' (in addition to existing sharding for commons and enwiki) [21:13:45] Logged the message, Master [21:13:47] Logged the message, Master [21:14:15] AaronSchulz: I think the deploy is finished. [21:14:23] I've tested the successful creation of images [21:14:26] haven't yet tested purging. [21:14:39] !log reedy synchronized wmf-config/InitialiseSettings.php 'Matched wikimania2013wiki configuration to that of wikimania2012wiki' [21:14:42] Logged the message, Master [21:14:42] ok [21:15:18] !log asher synchronized wmf-config/db.php 'returning db45 to service' [21:15:22] Logged the message, Master [21:19:17] AaronSchulz: purging tested successfully. [21:20:03] :) [21:20:05] * AaronSchulz was installing live headers but didn't want to restart FF since CZ would restart too [21:23:55] !log aaron synchronized wmf-config/CommonSettings.php 'Added swift backend config; unused atm.' [21:23:58] Logged the message, Master [21:27:02] AaronSchulz: just fyi, I also pinged in #wikipedia-de to let them know about the change and ask for pings if stuff goes weird. [21:29:16] maplebed: you know that it is nearly midnight here? [21:29:16] nice [21:29:22] !log aaron synchronized wmf-config/swift.php 'Added new thumbnail purge/import hooks handlers that use the swift backend class; unused atm.' [21:29:26] Logged the message, Master [21:30:13] DaBPunkt: yeah, I do... but I figure there'll be enough folks around to see radically broken shit. [21:31:37] besides, it wouldn't surprise me at all to find out that midnight is prime wikipediaing time. [21:31:39] ;) [21:32:12] It was just one time, ok >.> [21:32:27] uh huh. [21:44:41] DaBPunkt: hey :) Did you receive the email I sent to the ts-admin mailing list? :) [21:47:19] !log aaron synchronized php-1.20wmf2/includes/filerepo/file/LocalFile.php 'deployed 43aa35016b03935b27d439afe9a6b3f1aad1aa8b' [21:47:22] Logged the message, Master [21:50:56] Snowolf: subject? [21:51:13] DaBPunkt: "Account re-activation request" [21:53:04] Snowolf: done [21:53:11] DaBPunkt: thanks a lot! [21:54:37] jeremyb: Toolserver had some issues on willow, recovered now. I didn't fix dbbot manually [21:54:45] it is back up [21:54:52] @replag [21:54:53] Krinkle: No replag currently. See also "replag all". [21:54:59] @replag all [21:55:01] Krinkle: [s1] db38: 0s, db36: 0s, db32: 0s, db59: 0s, db60: 0s, db12: 0s; [s2] db52: 0s, db53: 0s, db54: 0s, db57: 0s; [s3] db39: 0s, db34: 0s, db25: 0s, db11: 0s [21:55:02] Krinkle: [s4] db31: 0s, db22: 0s, db33: 0s, db51: 0s; [s5] db35: 0s, db45: 0s, db44: 1s, db55: 0s; [s6] db43: 0s, db47: 0s, db50: 0s; [s7] db37: 0s, db56: 0s, db58: 0s, db26: 0s [22:03:23] AaronSchulz: are you guys done with your swift sharding stuff deployment? [22:03:45] awjr: yes. [22:04:02] cool thanks maplebed [22:06:10] yep [22:12:40] !log awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7' [22:12:43] Logged the message, Master [22:13:13] !log awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd828a8196d8bc877afdbd1559e8e6d639b51cef7' [22:13:17] Logged the message, Master [22:13:52] !log awjrichards synchronized wmf-config/CommonSettings.php 'bumping MobileFrontend resource version number' [22:13:55] Logged the message, Master [22:20:22] gn8 folks [22:34:08] !log awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile url transformation on testwiki' [22:34:11] Logged the message, Master [22:38:10] !log awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/stylesheets/sections.css 'Live hack to live test broken interface on ICS devices on very large articles' [22:38:13] Logged the message, Master [22:47:50] !log Fixed permissions in /h/w/conf/httpd by running find -group wikidev -not -perm 020 -exec chmod g+w \{\} \; [22:47:53] Logged the message, Mr. Obvious [22:53:47] !log Actually fixed it now with chmod -R g+w /h/w/conf/httpd [22:53:51] Logged the message, Mr. Obvious [23:00:35] Tue May 8 22:14:01 UTC 2012 mw27 commonswiki LocalFile::delete 10.0.6.41 1205 Lock wait timeout exceeded; try restarting transaction (10.0.6.41) UPDATE `site_stats` SET ss_images=ss_images-1 [23:01:43] brion: maybe I should also push that out of the main transaction [23:02:02] it's been how it is for a while, but that obviously causes lots of lock timeouts [23:02:21] it's like 99% of dberrors.log ;) [23:02:25] eek [23:02:36] yeah site_stats update should be outside if you can [23:03:15] though inside sounds better for correctness reasons, that'd mean we'd actually have to handle the rollback more usefully than by giving up :D [23:04:04] brion: Your "correctness reasons" would essentially mean "if we can't increment ss_images, we should rollback the upload" AFAICT [23:04:19] Forgive me if I say that doesn't entirely sound reasonable to me : [23:04:21] * :) [23:04:22] since we don't move back the file, rollback due to site_stats is lame [23:04:41] I guess a savy user would know to just delete again [23:05:11] of course if we stored files properly this wouldn't be such an issue... [23:05:17] Or we could take the risk that the counter wouldn't be decremented [23:05:35] RoanKattouw, in theory we want the counters to be accurate :) [23:05:38] the counter isn't a huge deal, and can be fixed with crons periodically [23:05:39] Which I think is actually less likely to happen if that operation isn't at the end of a big transaction [23:05:44] in practice we'd rather the upload gets saved [23:05:48] Yeah [23:06:22] know should I put it before or after the purge [23:06:24] *now [23:06:27] how long do we keep transaction open for upload? [23:06:48] * AaronSchulz leans towards after [23:06:58] Yeah make it after [23:07:05] site_stats is probably still prone to lock contention [23:07:08] stats is the lowest importance thing [23:07:17] So attempting to do the -=1 might take a while or throw an exception [23:07:22] In the latter case the purge would never happen, which is bad [23:08:06] ss_ on image is important, since plwiki uses it to make sure # of images stays at level 0 :) [23:09:09] RoanKattouw: I wonder if some hog is doing site_stats update and *then* a bunch of stuff somewhere... [23:09:16] and committing afterwards [23:10:01] I wonder if we have logging for this kind of thing? [23:10:11] It'd be interesting to see what actually ends up being grouped in transactions in practice [23:10:24] * AaronSchulz looks at binasher ;) [23:10:34] !log awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663' [23:10:37] Logged the message, Master [23:11:26] !log awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ '2b1e8573fdbcab0feb3a2481167b68fb96abf663' [23:11:29] Logged the message, Master [23:11:43] !log awjrichards synchronized wmf-config/CommonSettings.php 'Bumping MobileFrontend resource version' [23:11:46] Logged the message, Master [23:22:52] !log awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version' [23:22:55] Logged the message, Master [23:25:04] !log awjrichards synchronized php-1.20wmf2/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473' [23:25:08] Logged the message, Master [23:25:35] !log awjrichards synchronized php-1.20wmf1/extensions/MobileFrontend/ 'd43f5f19ff3599f16200d247b6838cfb04ef1473' [23:25:38] Logged the message, Master [23:25:55] !log awjrichards synchronized wmf-config/CommonSettings.php 'Bumping mobilefrontend resource version again' [23:25:58] Logged the message, Master [23:35:41] !log i'm gay [23:35:44] Logged the message, Master [23:42:58] RoanKattouw: can you kickban in ops as well ? [23:43:28] ChanServ You are not authorized to perform this operation. [23:43:34] * RoanKattouw isn't an op in osp [23:43:35] *ops [23:44:37] !log synchronized payments cluster to r115155, DonationInterface ccfbb304 [23:44:40] Logged the message, Master