[03:20:17] how can i force certain portlet to be loaded expended by default for anons? [03:20:28] particularly the toolbox [07:40:23] Downloading refs/changes/07/14007/1 from gerrit into review/tim_starling/work [07:40:24] error: unable to resolve reference refs/heads/review/tim_starling/work: Not a directory [07:40:24] fatal: Failed to lock ref for update: Not a directory [07:40:33] Anyone have any ideas why this happen? [07:54:47] vvv: can you do it again with verbose mode and paste that somewhere? git-review -d 14007 -v [07:58:02] hashar: http://dpaste.org/B8pxp/ [08:00:09] vvv: doh, that does not make any sense to me :( [08:00:45] is the review/tim_starling/work existing locally? Might want to delete that local branch if it does [08:01:54] No [08:02:05] There was "work" branch, but I deleted it [08:02:07] Oh wait [08:02:15] There's "review" branch as well [08:02:53] hashar: fixed by deleting the review branch [08:02:54] Thanks! [08:05:22] vvv: I think I fixed it upstream, you might want to update [08:05:42] hashar: fixed what and when? [08:06:12] vvv: your bug, I am not sure [08:06:28] I know that at some point git-review did not reused a branch already existing [08:06:38] I last updated my git-review about half an hour ago [08:06:38] so you got an error message when redownloading a change [08:06:50] so that might be another issue [08:06:57] hashar: I was not redownloading, I previously had a real branch called "review" [08:07:08] 9c349f2 - fix -d not reusing already existing local branch (3 months ago) [08:07:11] ahhh [13:34:34] <^demon> Krinkle: To answer your question on "when rebase is useful if there's no conflicts" -- https://gerrit.wikimedia.org/r/#/c/9623/ [13:35:03] <^demon> The dependency was updated to a new patchset, so now the parent commit on 9623 is invalid and needs rebasing. [13:35:39] Ah, right [13:35:52] but rebased on master or rebased on that change set? [13:36:00] the latter, right? [13:36:28] Interesting.. gerrit continues to make the link from changset to change set, but Git obviously uses the commit hash as parent. [13:36:29] <^demon> In this case, the latter. [13:37:00] <^demon> Right, git needs a sane parent, but gerrit still keeps track of the dependency so it doesn't magically disappear (and you shoot yourself in the foot by missing a dependency you really needed) [13:37:28] yeah, so this way it only allows merging of revisions that have a parent that is also reviewed (obviously) [13:38:46] ^demon: btw, maybe you know a solution to this [13:39:06] https://www.mediawiki.org/wiki/Git/Workflow#Create_a_dependency [13:39:11] I wrote that section [13:39:27] ^demon: Is there a way to use git review instead of git push refs/for/ syntax? [13:39:39] I never remember the latter.. [13:39:40] <^demon> Not that I know of, no. [13:39:52] (since git review will try to push both) [13:40:31] <^demon> I'm not terribly familiar with git-review. I don't personally use it often other than `git review -d [num]` because it's a timesaver. [13:41:14] ^demon: you don't use plain 'git review' to push a change for review? [13:41:29] <^demon> Nope. [13:41:44] So you always use git push gerrit HEAD:refs/for/master ? [13:42:10] <^demon> Yup, got in the habit. [13:43:17] k [13:43:52] <^demon> git-review only works over ssh, and I've got some outside gerrit repos other than ours that push over https. [13:43:57] "MediaWIki has 19 heads" [13:44:01] muahaha [13:44:55] <^demon> Krinkle: gerrit's own gerrit doesn't use ssh; ;-) [14:03:19] ^demon: heh [15:26:56] Hi. [15:29:02] Code review systems with pre-commits workflow are nice. Following the barkeep discussion, I played a little with phabricator on my FreeBSD server and I've seen the arcanist script invoked /bin/bash instead to find where bash is with /usr/bin/env bash. At 11:23 am UTC, I've submitted a patch (create a git branch, do the changes, "arc diff", fix the config, "arc diff", fix another point of the config, "arc diff", hop it's published). At · D29 [15:29:17] At 1:35 PM, hop the proposed patch is merged. [15:30:30] I've the feeling code review system really eases the patch from external people acceptance workflow. [15:50:09] New patchset: Hashar; "ant: make git macros to always use 'dir'" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14058 [15:50:10] New patchset: Hashar; "ant: git-reset-hard target" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14059 [15:50:10] New patchset: Hashar; "ant: fix git-clean-dfqx target" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14060 [15:50:11] New patchset: Hashar; "ant: make `git` fail on error" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14061 [15:50:12] New patchset: Hashar; "ant: redo clone dependency, import-project target" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14062 [15:50:13] New patchset: Hashar; "ant: import-extension helper, import MediaWiki ext" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14063 [15:50:13] New patchset: Hashar; "ant: targets for core and Wikidata projects" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14064 [15:52:14] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14064 [15:52:15] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14063 [15:52:16] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14062 [15:52:17] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14061 [15:52:18] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14060 [15:52:19] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14059 [15:52:20] Change merged: Hashar; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14058 [15:52:44] ^demon|away: the Jenkins stuff I was talking about is above :-] [15:52:57] ^demon|away: would let us easily clone in various jobs [16:02:38] ^demon|away: and next gerrit trigger is going to be able to react to "change-merged" event !!! [16:02:39] | * f10925e - WiP: Add support for triggering jobs on the change-merged event (5 months ago) [16:03:34] I am off for today [17:25:48] @help [17:25:49] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.2.4 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [17:25:55] shepazu: ^ [17:26:00] thanks [17:27:59] Hello matthiasmullie RoanKattouw chrismcmahon kaldari bsitu: Look forward to deploying AFT5 together today! Has everyone completed their code review for today's deployment? [17:28:19] Aaron and I just talked about Mathias's core change (for the contributions page) [17:28:41] I'm splitting it up in 3 parts now [17:28:52] RoanKattouw: Cool. What are your findings on MyContribs? [17:29:26] Aaron doesn't like the way the core change is done very much, he'd rather we'd have a class that lets you combine pagers [17:29:44] So I think the compromise we've reached is that I'll split out the scary part, deploy it, then clean it up later today by writing that multipager class [17:30:31] (confirmed with Aaron that that's accurate) [17:32:12] RoanKattouw AaronSchulz : Sounds like a reasonable plan for MyContribs. matthiasmullie : Is this something you can do easily? How long would it take? [17:32:30] I'm the one doing these things [17:32:32] Working on them right now [17:33:52] RoanKattouw: Thanks so much for doing this! If you encounter serious issues, we would be prepared to hold back on MyContribs, so we can push everything else. Keep us posted! [17:35:58] matthiasmullie: While Roan is working on MyContribs, are there any critical code reviews that are still needed from other team members? Or are we good to go? [17:36:42] I just emailed kaldari & bsitu :) [17:37:32] there's 4 more in total (of which 1 not particularly critical) [17:40:07] preilly: Are you deploying? [17:40:19] RoanKattouw: not yet [17:40:43] RoanKattouw: not until 3pm-4pm PDT [17:40:47] OK [17:44:09] preilly: Wait, did you just merge something that's not supposed to be deployed yet into 1.20wmf5? [17:44:24] RoanKattouw: yes [17:44:34] RoanKattouw: it's totally okay if it goes out early [17:44:36] There are other deployments on the calendar before yours [17:44:38] Oh OK [17:44:44] RoanKattouw: it's a critical bugfix [17:45:04] matthiasmullie: Thanks for the update. kaldari and bsitu, please let us know if you encounter any serious issues with any of your AFT code reviews. [17:54:51] OK the splitting is done, right now I'm just waiting for Jenkins to finish running the tests on the last one of those three so I can merge it [17:59:20] Thanks, RoanKattouw ! What is the impact of removing the scary part of MyContribs? Does this mean that we would expose oversighted comments to the public? If so, we should plan to fix this today, or revert at end of day, because we are going to get some serious flack if offensive comments are shown to the public. [17:59:35] Wait, why would THAT happen? [17:59:57] I'm about to merge that code in, everything's fine [18:01:03] fabriceflorin_: that's in a different (AFTv5) commit; Roan's splitting up the core-part [18:01:20] as long as the AFT commit gets in, there shouldn't be any problem [18:01:25] RoanKattouw: I don't know if it would happen, but this could be related to our last requests on Bugzilla: https://bugzilla.wikimedia.org/show_bug.cgi?id=36773 [18:01:55] matthiasmullie: I split off two parts of https://gerrit.wikimedia.org/r/#/c/13880 and left the rest alone, now waiting for jenkins-bot to +1 it so I can merge [18:02:08] matthiasmullie: Thanks for clarifying that there should not be a problem. I just wanted to be sure, so we don't do something bad that would cause serious issues with the community. [18:08:54] To be 100% clear, what we don't want to see happen is for hidden or oversighted comments to be shown on MyContribs. This is now working properly on prototype, from a user interface standpoint. All the obscenities I used to test the AbuseFilter are correctly hidden on MyContribs at this URL: http://prototype.wikimedia.org/release-en/Special:Contributions/Fabrice_Florin [18:11:43] Hello yoni_omniti, good to see you here today! Thanks again for all the good work that you and the OmniTI team did to get Article Feedback up to the stage that we could start a wider deployment today! We miss you guys already ;o) [18:12:01] You are welcome :) [18:13:38] matthiasmullie: What's the status on when you think we can start deploying AFT to test.wikipedia.org ? Are there any show-stoppers that we should consider reverting so we can start deployment? [18:13:56] rawr Jenkins is broken [18:14:04] That's why it's not confirming my commit [18:15:00] RoanKattouw: Hmmm. That seems sad. Do we need Jenkins to approve your revisions? Or can we approve them manually for now? [18:15:09] I can approve it manually [18:15:20] Or ... nm [18:15:22] I just unstuck it [18:16:04] The test result should come through in a few minutes [18:16:08] RoanKattouw: Very cool. I love it when things get unstuck -- well most of the time, that is ;o) [18:18:31] fabriceflorin: there's a few more commits awaiting review, bsitu & kaldari should be working on them :) [18:18:44] as in "reviewing them", that is [18:22:46] Cool. bsitu and kaldari are reviewing now, but so far they don't think anything needs to be reverted. Fingers crossed ... [18:29:57] Sorry I'm missing the deployment. :-) [18:40:14] RoanKattouw: What did Jenkins say about the MyContribs revisions and your split? Did we pass the audition? [18:40:21] Yeah it passed [18:40:39] Jenkins just got stuck for a half hour for some reason, then when I unstuck it it approved the commit in 3 minutes [18:45:02] Very cool! matthiasmullie, are there any final revisions awaiting approval, or are we good to go? (we are more than half-way through today's deployment window, so we should plan to start pushing code very soon). [18:46:06] I only saw one unreviewed commit, but it was a minor translation thing [18:46:42] https://gerrit.wikimedia.org/r/#/c/13885/1 [18:47:12] All Kaldari is asking is that you not change the keys. But I think we can roll without this, fabriceflorin ? [18:49:27] Sounds good to me. I see that Matthias just uploaded patch set 2, which hopefully fixes the problem. If not, I would recommend deploying anyway, because we are running out of time, and we will need a lot of time to test all these features. [18:49:29] that was indeed the only one left (updated now though) and I'm now getting everything up to testing [18:50:07] matthiasmullie: Great news! I am preparing a batch of links for us to test when we are on testing. [18:51:30] It's been merged by Roan. I think that's it from the deploy plan [18:52:02] matthiasmullie: you're good to go I think [19:23:22] fabriceflorin: FYI: the code has been pushed to test [19:23:29] I am now rebuilding message cache [19:23:41] and still need to verify that _all_ code has gotten up there [19:24:00] but you could get started testing, I suppose :) [19:25:24] matthiasmullie: Wonderful! I just sent you all an email, with links to check on test.wikipedia.org. Onward! [19:26:48] something seems not ok (both AFTv4 & AFTv5 are showing on golden sparrow) [19:28:11] matthiasmullie: Yes, I noticed. Were you able to decrease the AFTv4 lottery on the test site? Any idea why this is not working? Anything I can test for you? [19:28:30] I'm looking into it as we speak :) [19:30:30] I cannot reproduce this issue for most pages on testing, except for these with the AFT5 category: http://test.wikipedia.org/wiki/Category:Article_Feedback_5 [19:34:59] fabriceflorin: I think I know what's causing the issue: testwiki doesn't seem to have the AFTv5 categories blacklisted (enwiki does) [19:35:02] I'll be updating config [19:35:12] any other issues so far? [19:35:46] OK sounds good. I am going through my list of URLs to check, which I just emailed to the group. No show-stoppers for now. [19:38:17] We still have the issue of the 'BETA' label showing up garbled on the feedback page, but that may be related to the fact you have to update the messages. [19:44:38] fabriceflorin: I believe the AFTv4/AFTv5 issue should be solved [19:45:26] (currently a quick-fix only on testing; committed config change as well, but needs review before I'm pushing it live - pinged RoanKattouw_away, but, he's away - anyone else I can bother?) [19:48:11] Yes, nice job! I can confirm that AFT5 now appears by itself on the pages where it is supposed to appear. [19:48:38] oddly, I'm not seeing AFT at all in IE8 [19:48:51] tried two different VMs, Vista and Win7 [19:49:24] matthiasmullie: RoanKattouw probably went out to lunch. Perhaps you could ask Bsitu or AaronSchulz for advice. [19:49:27] * chrismcmahon wonders if Yoni's very last change made it to gerrit and/or test [19:50:46] yes, it did [19:51:14] chrismcmahon: Thanks for bringing up the IE8 issue. yoni_omniti : Are you able to reproduce the IE8 issue reported by Chris? [19:56:33] now I'm seeing aftv5 in IE8 \o/ on test [19:58:14] matthiasmullie: OK, I have gone through my list of URLs to test and do not see any serious issues, so I think we are good to go to production, from my standpoint. [19:58:16] aw crud, and getting the "Object doesn't support this property or method" error from load.php like we did before Yoni's final fix [19:58:29] matthiasmullie: Other than the IE8 issue reported by Chris, everything works well for every feature I checked. My main findings are that the Talk page link is visible to all users, including readers (should be only editors) -- and that the Beta label still appears garbled on the feedback page (<articleFeedbackv5-beta-label>). [19:58:56] posting feedback doesn't work at all in IE8, causes the error above, that Yoni fixed before [20:00:44] chrismcmahon: Unless matthiasmullie finds that the Gerrit ticket for IE8 was not included, I think we should go ahead with deployment despite the IE8 issue, because IE8 has been unsupported for a while now. But if it's easy for matthiasmullie to remove it from our list of supported platforms, that would be great. [20:01:32] fabriceflorin: sorry i missed that, was in another window [20:01:36] what is the ie8 issue? [20:01:51] I am concerned that it is 1pm PT already, which is the end of our deployment window, and it will take us at least an hour to go to production and test it. So I am willing to live with no IE8 for now, if that's the price we pay. [20:02:30] chrismcmahon: could be that my change didnt get to gerrit. i will verify [20:03:00] indeed - it didnt get to gerrit [20:03:05] Hi yoni_omniti , how is IE8 working for you on test: https://test.wikipedia.org/wiki/Golden-crowned_Sparrow - chrismcmahon reported that he could not see the feedback form at all, then he said he could see it, but that it would not let him post. Can you reproduce? [20:03:44] matthiasmullie: Can you include the IE8 change in your code before deploying to production, please? We tested it on prototype, so it should be fine .... [20:04:14] hooray for regression testing :) [20:05:37] Yoni, can you please post the Gerrit ticket URL here, for the record? Are there any other Gerrit revisions from OmniTI that fell through the cracks, from what you can tell? [20:06:31] yoni_omniti: where can I find the commit? [20:06:50] give me a second to sort out git and do git review [20:12:23] matthiasmullie: https://gerrit.wikimedia.org/r/14104 [20:12:36] that fixes the ie8 issue [20:13:27] fabriceflorin: unfortunately i dont speak gittish too well. only place to look for remaining commits would be the omniti branch [20:15:31] Thanks, yoni_omniti , much appreciated! matthiasmullie does this give you enough info on how to fix the problem? Do you need more folks to do code review for his ticket? [20:15:51] its a 2-line change, one of them is comment :) [20:16:13] I reviewed it [20:16:25] I'll be pushing it to test right away [20:16:29] thanks [20:16:34] i have to go afk for a few minutes [20:16:38] Cool, thanks, matthiasmullie ! [20:16:55] Hello rsterbin , good to see you here today! [20:16:59] hey [20:17:39] matthiasmullie: https://gerrit.wikimedia.org/r/#/c/13549/ is in need of review, but it doesn't introduce any new functionality [20:18:48] Thanks, rsterbin -- I think this deployment is going well. We are about an hour behind schedule, but all the features are testing well so far. All we have to do now is check the IE8 fix. Next stop: production! [20:18:55] woot [20:25:08] still seeing ie8 error, matthiasmullie let me know when that should be changed? [20:25:31] chrismcmahon: busy getting it up there; should be there soon, I'll let you know :) [20:26:54] chrismcmahon: it's pushed now, could you retry? [20:27:05] checking... [20:27:08] fabriceflorin: FYI: there was something wrong with the beta-label; fixed it as well [20:27:46] im back [20:27:54] matthiasmullie: Cool, thanks! I can confirm that it was fixed. [20:28:17] IE8 is OK [20:29:33] fabriceflorin matthiasmullie ^^ move off testing now? [20:30:09] chrismcmahon: Excellent news, thank you all! Yes, we are ready to push to production now, unless anyone has any final objections. [20:31:02] it's ok for me [20:33:38] matthiasmullie: OK, sounds good. Let's go ahead with this, if you are comfortable pushing through for another hour or two. [20:33:55] pushing to production ;) [20:35:10] All right! I am sending you all an email with links to check when the code has been pushed to en-wiki. I will be away from my desk for 15 mins. now, to go grab a bite to eat. See you shortly … [21:04:12] fabriceflorin: chrismcmahon: yoni_omniti: rsterbin: everything's on production [21:04:16] get your testing gear on ;) [21:04:29] OK, I'm on it! [21:05:07] woot [21:06:16] check :) [21:09:39] fabriceflorin, matthiasmullie: when posting my >255 comment, the confirmation message was missing on the cta [21:10:17] CTA 4 is appearing outside of the gray box [21:11:31] rsterbin: what browser is that? I'm not seeing anything weird on firefox [21:12:09] this is ff14 [21:12:12] on ubuntu [21:12:49] i'll send a screenshot [21:15:02] I was also able to see some similar formatting problems with CTAs earlier, but only on Chrome incognito -- seemed related to the absence of a cookie, because I couldn't reproduce on other platforms. I have screenshots. So I didn't report them then, but will now that you are finding them elsewhere. [21:17:53] fabriceflorin -- i've sent the screenshots (one for the display bug, and one for the missing message) [21:18:13] the formatting on the other ctas looks fine for me [21:18:19] rsterbin: Thanks! much appreciated. [21:18:46] i double-checked with my testing ff profile (no extensions, default installation, anon user), and see the same issues there [21:23:01] fabriceflorin: is this something you can live with on production this week? [21:23:44] I'll be looking into it tomorrow [21:24:26] and does anyone encounter any other lurking issues? [21:25:32] IE8 seems to be fine, no issues to report, and I've looked at FF12 as well [21:26:25] rsterbin: thanks for the screenshot, can't reproduce atm but I know where to look now ;) [21:26:34] I am about half-way down my list of things to check, but it is looking very good for now. [21:27:42] I could not reproduce issues with the 255 character comments. Both new comments and old comments appear fine, in both permalink and feedback pages. [21:29:58] fabriceflorin: you wouldn't, unless you got cta5 [21:30:11] it's related to the cta, not the length of the comment [21:30:29] I am getting some weird error messages (both on Meta and Office wiki) that I've never seen before upon submitting a revision: "A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was: (SQL query hidden) from within function "SqlBagOStuff::set". Database returned error "1637: Too many active concurrent transactions (10.0.6.50)"." – is this a known issue? [21:31:07] waitaminnit, think I have an issue... [21:32:27] matthiasmullie: PHP fatal error at http://en.wikipedia.org/wiki/Special:ArticleFeedbackv5 [21:34:20] wow [21:34:24] rsterbin: Yes, you are correct about the CTA5. I can see an issue on my end as well, on both Firefox and Chrome, where the 'Thanks, your feedback was posted here' is missing, no matter how long your comment is: http://en.wikipedia.org/wiki/Golden-crowned_Sparrow?aftv5_cta=5 [21:34:38] chrismcmahon: wow - that's odd; not the case on test [21:34:43] i'll look into it [21:35:10] wow, that's a new one [21:35:50] Yes, the central page issue is serious. I wonder if RoanKattouw has any ideas about what could be causing this. http://en.wikipedia.org/wiki/Special:ArticleFeedbackv5 [21:37:00] dartar: I am assuming the error messages you get on meta and office do not appear related to AFT. Right? [21:37:14] yeah, sorry - nothing related to AFT [21:37:23] ops is on it [21:37:45] Cool. Just wasn't planning on checking our impact on edits quite yet ;o) [21:47:15] chrismcmahon: I can confirm that your oversight tests are appearing along with mine on the Suppression log, with the proper format. When testing oversight features, I recommend adding a comment that this is for testing purposes only, so we don't freak out the oversighters ;o) [21:48:02] matthiasmullie: The Central Feedback page is now back up, and seems to be working for me -- maybe it was just a temporary burp related to other issues: https://en.wikipedia.org/wiki/Special:ArticleFeedbackv5 [21:48:41] fabriceflorin: yeah, noticed as well; didn't touch anything though, so not really sure what happened yet [21:48:55] fabriceflorin: got it, thanks [21:49:21] It's probably an ops issue, though we should keep an eye on it. Only a few more links to check at this point, overall we seem to be in good shape. [21:50:36] fabriceflorin: I got the error message from Roan & the issue occured in AFT code [21:51:05] however, backtracking the code, couldn't spot what might cause the issue at first sight [21:51:59] not sure what's up yet, and not sure if we should be worried about it [21:52:08] Tim-away: What happened to fatal.log? [21:52:14] Hmmm. What was the error message? Do we know if it was due to an individual user action? [21:52:39] RoanKattouw: are you on the ops list? [21:52:49] Oh right [21:52:52] * RoanKattouw searches [21:53:33] Reedy also knows about it [21:53:58] I have to go for an hour or so [21:54:04] for the record: PHP fatal error in /usr/local/apache/common-local/php-1.20wmf6/extensions/ArticleFeedbackv5/ArticleFeedbackv5.render.php line 1327: [21:54:04] Argument 1 passed to ArticleFeedbackv5Render::getActivityCount() must be an instance of stdClass, string given, called in /usr/local/apache/common-local/php-1.20wmf6/extensions/ArticleFeedbackv5/ArticleFeedbackv5.render.php on line 1012 and defined [21:54:48] OK off the phone now [21:54:51] Let's find that log file [21:55:05] fatal.log is empty [21:55:34] l-wx------ 1 nobody nogroup 64 2012-07-03 21:55 4 -> /var/log/mw/fatal.log-20120628 (deleted) [21:56:11] Oooooh wait a minute [21:56:20] Is this a consequence of me having set up logrotate for this? [21:58:30] chrismcmahon: Thanks for sharing the PHP fatal error message. matthiasmullie or RoanKattouw : Does this message mean anything to you? [21:58:39] matthiasmullie: I have a backtrace http://pastebin.com/hVMLqRuW [21:58:48] Sorry it took so long, I was on the phone and the backtrace logging mechanism was broken [21:59:46] RoanKattouw: Thanks for the backtrace! I also remember you saying earlier that you were going to work on something to fix the MyContribs later today. Could this fatal error be related to that in any way? [22:00:24] No, this is unrealted [22:00:26] *unrelated [22:01:24] Got it. Just thinking out loud. I guess I better keep quiet and let you and Matthias try to isolate the source of this fatal error, since I can't do much on my end. [22:01:41] No worries [22:03:22] What I can report is that I went through my entire check list and have no other issues to report, besides what Reha already reported about the CTA 4 and 5 formatting issues, plus the earlier issue I had with the Talk page link being available to everyone (which I can live with for a while). [22:06:03] And I don't think the CTA problem are too worrisome, since they still work functionally, it's only a formatting issue. We could change the bucketing, but I don't think it's worth it. I would rather we focus on finding the source of the fatal error. But I will open a bug for these CTA issues, using Reha and my screenshots. [22:10:50] RoanKattouw: fatal.log has some tricks Tim sent an email about it in ops list [22:11:02] New patchset: Krinkle; "(bug 36143) Improve testswarm-submit job name" [integration/testswarm] (master) - https://gerrit.wikimedia.org/r/14156 [22:11:04] RoanKattouw: you are probably aware of it though :) [22:11:54] Yeah I just re-read it [22:11:57] Change merged: Krinkle; [integration/testswarm] (master) - https://gerrit.wikimedia.org/r/14156 [22:12:05] It was sent while I was at a conference [22:12:21] I restarted the logger and that fixed it [22:12:31] I am not sure why we have a dedicated logger instead of plain syslog [22:12:56] Or udp2log or whatever [22:13:06] Krinkle: is that testswarm submit for the labs instance ? [22:13:15] not eyt [22:13:17] not eyt [22:13:22] not yet [22:13:38] ko tey ton [22:13:46] my reverse is bad [22:13:47] hashar-away: is that deployed automatically from puppet ? Or do I need to someone else to do something? [22:13:56] need to be manually deployed on gallium [22:14:10] can you do that? [22:14:17] if you can test it right away yes [22:14:32] sure, I'll trigger jenkins now [22:14:37] (after you say the word) [22:15:13] ahh that is a submodule ! [22:15:31] oh, so I need to update something else? [22:15:51] grrr. I wish submodules could just be set to master/HEAD [22:16:39] hashar: right ? [22:21:40] New patchset: Krinkle; "Update submodule: integration-testswarm" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14157 [22:23:17] matthiasmullie RoanKattouw : Any more ideas on what might have caused the fatal error? Should we be worried that it might happen again? If so, is there anything you think we could do about it? [22:23:35] Krinkle: got disconnected sorry. [22:23:55] fabriceflorin: Something in AFT is doing something wrong, Matthias should investigate [22:24:12] I have him the error message and (later, after wrangling log files) a backtrace, he should be able to figure it out from there [22:24:18] I have to go to the dentist in a few minutes [22:24:22] New patchset: Krinkle; "Update submodule: integration-testswarm" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14157 [22:24:32] hashar: "Project policy requires all submissions to be a fast-forward." [22:24:33] intersting [22:24:52] fabriceflorin: I can't reproduce the error anywhere (test, proto, local) [22:25:09] but I'm working on something that I think/hope will fix the problem [22:25:28] New patchset: Hashar; "update testswarm fetcher" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14158 [22:25:47] Krinkle: yeah I have set up the integration/jenkins repo to enforce fastfoward only. To avoid the mess caused by gerrit merges [22:25:47] RoanKattouw: Thanks. Do you think we should consider reverting? Or just keep an eye on it? It's getting pretty late for matthiasmullie out there in Belgium, he may start to fade ... [22:25:49] Change abandoned: Krinkle; "(no reason)" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14157 [22:25:53] I can no longer repro it on enwiki, but it was consistent there for a bit [22:25:58] hashar: ok [22:26:12] It's possible that the db40 hiccups are causing edge cases in AFT to be hit I guess [22:26:23] Change merged: Krinkle; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/14158 [22:26:29] matthiasmullie: Keep us posted on your proposed solution. [22:27:14] RoanKattouw: What are the db40 hiccups? Is this an unrelated problem potentially affecting AFT? Or is it AFT's problem? [22:27:40] Most likely unrelated [22:28:08] But I could see that or some other intermittent issue causing AFT to get data back that it doesn't normally receive and choking badly on that [22:28:15] Just a theory though, I haven't examined the code [22:29:23] Hmmm. Not sure what we can do about the db40 problem. Did it just start recently? If so, would it have affected the earlier version as well? I am trying to figure out is we should consider reverting or not. [22:30:43] I don't think it's related [22:30:45] You could ask ops when it started [22:30:49] Actually it might already be fixed [22:32:37] Alright dentist time [22:51:09] fabriceflorin: potential fix under review atm [22:52:09] Thanks, Matthias, good for you! Who is reviewing your fix right now? [22:53:00] bsitu is on it [22:53:24] not sure if the "fix" will actually fix the problem, as I can't reproduce it, and it's more of a guess at what's wrong [22:53:44] if not, I'm afraid we'll have to revert [23:01:49] matthiasmullie: Thanks. I just sent an email asking tewwy (Terry Chay) to make a decision about what to do next, since this is outside of my area of expertise. I cc: d you all on that message. [23:02:36] matthiasmullie: I'm just guessing but that looks like an unhandled exception. getActivityCount() called for something, probably from the database, and got back a string (probably an error message) instead of an object? [23:03:21] that's basically it, yes [23:03:47] first though: fabriceflorin: chrismcmahon: potential fix is on production [23:03:55] so far, I don't see the error [23:04:03] let's see if it stays that way [23:04:14] chrismcmahon: indeed, instead of an object, it received a string [23:04:32] matthiasmullie: Thanks for burning the midnight oil to come up with this fix! What do you think is the best way for us to test it? [23:04:33] which should never be the case, according to the code [23:05:03] fabriceflorin: I'd just test central feedback page & feedback page for some articles, and try again in a short time [23:05:19] fabriceflorin: if central page is OK, there's not much else we can do [23:05:23] if we can't seem to trigger the error again, I'm pretty confident that the issue is solved [23:06:08] chrismcmahon: however, fetching the data from the db is done in 2 queries; 1 which basically only fetched the ids; another which then fetched the additional data and just squishes it in the array containing ids at that moment [23:06:57] fabriceflorin matthiasmullie the bottom line is that AFT code received a response from essentially Mediawiki that it never expected to receive, and handled it not so well. [23:07:14] apparantly, _it_ didn't always return new data for all entries (some replication lag? not sure - have to investigate tomorrow), so the array didn't get enriched with data for all entries, and some only remained _id_ (rather than all the new data) [23:07:21] hard to anticipate that sort of thing [23:07:48] that's what it looks like [23:08:02] and so far, the fix seems to be holding up, afaict [23:08:24] Thanks, matthiasmullie : I will bang on central feedback page now and see if I can do anything useful. [23:08:35] please do :) [23:17:00] heatherw: have a button for you in a minute [23:17:14] awesome! thanks!! kaldari [23:17:55] possibly... still need to figure out a way to do it... [23:18:24] heh [23:20:41] heatherw:

[[Article]]

[23:22:19] awesome!! [23:23:51] drecodeam: Howdy. I'm going to try to look at your new UploadWizard code soon (definitely by the end of the week). [23:24:28] kaldari: ya... i would be done by uploading from a photoset also by then i guess [23:24:45] awesome [23:24:47] although there would still be minor issues with it... [23:24:54] that will be a nice feature [23:25:21] Also kaldari, I am running late according to my timeline in my proposal [23:26:00] because of the Berlin Hackathon and another conference I went to after that, I would not be able to complete my testing before the midterm evaluations [23:26:38] What day are the evals? [23:26:56] but I have also done a major chunk of the geolocation support, most of which I completed in the Berlin Hackathon itself, so I guess the time would be balanced out there [23:27:06] evaluations are from 9-12th of this month [23:27:33] drecodeam: Yeah, I imagine it will balance out in the end [23:27:57] Also, you have a pretty ambitious project schedule to start with :) [23:29:07] I spent quite a lot of time in understanding the code and all the classes in it, so now its pretty smooth in writing the new code [23:30:24] that's good. Most of the work is just understanding all the existing code [23:30:38] since there's a lot of irt [23:30:40] it [23:31:25] Ya, and it's pretty beautiful code :) [23:31:48] I have learnt a lot from it, still learning actually ! [23:32:22] kaldari: Also, what should we be using for the testing ? [23:32:32] it's a bit convoluted in places, but hopefully understandable [23:32:56] Jasmine, qunit or anything else.. since I have not worked on any of them, I have to learn them from the scratch [23:33:25] I think the Flickr part is going to be tricky to test... [23:33:42] matthiasmullie: I have been banging on the central feedback page for a half-hour now, and cannot get it to malfunction. I filtered the page every which way, and took about 50 different actions, mostly hiding inappropriate posts: https://en.wikipedia.org/w/index.php?title=Special:Log&offset=&limit=50&type=articlefeedbackv5&user= [23:33:48] are unit test required? [23:34:25] fabriceflorin: it constantly seems to be working fine here as well, whereas before it broke on multiple ocasions [23:34:35] I am not very familiar with testing, and I haven even started researching on it right now. So I would not be able to comment at it right now [23:34:42] I think it's safe to assume we indeed fixed the issue [23:34:52] Sounds one-time. [23:36:10] kaldari: I guess we can discuss testing next week, till that time I would also learn more about it, and I would also be done with the Flickr code [23:36:18] matthiasmullie: tewwy is back at his desk now, reviewing all this. Can you past the Gerrit ticket you just posted and that Benny approved, so he can review? [23:36:30] tewwy: what do you mean? [23:36:35] yeah, sure [23:37:02] tewwy: https://gerrit.wikimedia.org/r/#/c/14159/ this was it [23:37:15] some more info: [23:37:38] drecodeam: sounds good [23:37:42] the data is fetched in 2 queries: basically 1 that fetches some ids, and one that fetches additional data [23:37:57]

I am not sure why we have a dedicated logger instead of plain syslog [23:37:58] matthias: It is also my impression that the fatal error cannot be reproduced now. I certainly pushed the central feedback page as much as I could and neither you, nor chrismcmahon nor myself can reproduce the problem. So that's looking promising. [23:38:12] after the first query, an array ($ids) is built with this data, with all ideas in the right order [23:38:17] syslog has a length limit, I think it is 1024 bytes [23:38:24] too short for a full backtrace [23:39:40] and its messages are single lines [23:39:44]