[00:50:22] !logs [00:51:07] morebots: help [00:51:07] I am a logbot running on wikitech-static. [00:51:07] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [00:51:07] To log a message, type !log . [00:51:18] descriptive! [00:52:01] Ryan_Lane: if I wanted morebots to join #wikimedia-fundraising -- do you know how to do that? [00:52:23] mwalker: hm [00:52:36] we just need to change the config, I believe [00:53:52] gn8 folks [00:54:24] mwalker: done [00:54:29] Ryan_Lane: yay thanks ;) [00:54:50] yw [06:08:36] Susan: thanks for the cc but I'm not interested in following. Cf. btw https://www.mediawiki.org/wiki/Thread:Talk:Echo_(Notifications)/Deployment_roadmap [06:09:17] Ah, thanks for the link. [06:09:29] The bug was originally just about Meta-Wiki, but I expanded it at the last minute. [06:09:33] That's why I cc'd you. [06:09:38] But no worries. [09:35:38] [3888eacc] 2013-05-17 09:35:20: Fatal exception of type MWException [09:35:41] on meta, just now [09:35:58] Ah, never mind, probably a Translate error. [09:36:40] yup [11:16:12] ^demon: morning! can you help me with a squid caching issue? there is some broken output stuck in there... [11:25:57] <^demon> I can have a look sure. [16:01:11] greg-g, "FIXME: July 4th deploy?!?!?!" what's wrong with a june 4th deploy? (https://www.mediawiki.org/?diff=690870 ) [16:01:25] july 4th* even [16:06:56] us holiday? [16:10:32] Krenair: Reedy yeah, us holiday. Most ops will be off [16:10:43] what? [16:10:47] i have a holiday? [16:10:51] mutante: not you [16:10:54] damn it [16:11:10] you're still German or something [16:11:11] Oh that's their independence day, isn't it... [16:11:17] Krenair: yep. [16:11:39] "National Defense Transportation Day" wth? [16:11:47] o_O [16:14:52] I love transporting defense [16:46:47] greg-g: So that gerrit change got merged. When do you want to turn CodeEditor for CSS and JS on in Beta and start telling people about it? [16:47:28] anomie: I'll tell people about it in today's deploy highlights email, which I'll forward to -ambassadors with it called out. [16:47:40] anomie: so, turn it on Monday? [16:48:31] greg-g: For beta? Works for me. Any idea how long you want to give it before starting to turn it on in production? Or should I just wait for the email? [16:50:44] anomie: Turn on in betalabs on Monday, then test it, then turn on everywhere on Wed? sound reasonable? [16:52:26] greg-g: Wednesday's no good, I'll be on an airplane to AMS. Besides, that might be too quick for some people. Remember the recent complaints about changes with no notice. [16:54:19] fair... [16:54:55] anomie: roll it out with wmf5, starting May 27th, done on June 5th [16:55:04] just ride the Core train [16:55:16] thought people were chomping at the bit for this [16:59:06] hello, some user reports that they see [d615a856] 2013-05-17 15:13:31: Fatal exception of type MWException on login [16:59:31] Reedy: I know you are/were around [16:59:43] I'll have a look [17:00:17] Aha [17:00:18] 2013-05-17 15:13:31 mw1081 zhwiki: [d615a856] /w/index.php?title=Special:%E7%94%A8%E6%88%B7%E7%99%BB%E5%BD%95&action=submitlogin&type=login&returnto=Wikipedia:%E4%BA%92%E5%8A%A9%E5%AE%A2%E6%A0%88/%E6%B1%82%E5%8A%A9&returntoquery=action%3Dedit%26section%3Dnew%26preload%3Dtemplate%253Asign Exception from line 1766 of /usr/local/apache/common-local/php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php: Bad user row looking up local user [17:00:18] 維基小霸王@zhwikiquote [17:00:23] Some people. But there's lots more people who don't know about it (yet). [17:00:27] Core train could work, although I'd have to look into the proper manner of progressively adjusting the config variable (I guess first default = false with true on test, test2, mw.org; then default=true and false on "wiki"; then add true for enwiki; then remove the false for "wiki"?) [17:00:39] Oh, Reedy is here. ^ [17:00:55] huh [17:01:12] wtf [17:01:16] Tab complete doesn't think you're in here [17:02:07] reedy@fluorine:/a/mw-log$ grep "Bad user row looking up local" exception.log -c [17:02:07] 11 [17:02:36] Logged it as a bug... [17:02:55] https://bugzilla.wikimedia.org/show_bug.cgi?id=48579 [17:03:04] thanks [17:06:25] Reedy: hey did you post that user's password on bugzilla? it says: [17:06:26] #1 /usr/local/apache/common-local/php-1.22wmf3/extensions/CentralAuth/CentralAuthPlugin.php(60): CentralAuthUser->attemptPasswordMigration('********') [17:06:26] #2 /usr/local/apache/common-local/php-1.22wmf3/includes/User.php(3470): CentralAuthPlugin->authenticate('???????????????', '********') [17:07:29] lol [17:08:56] sorry I guess I should privmsg Reedy, instead of saying it here and attract more people [17:09:47] liangent: I suggest you switch the bug in question to security, if you have the access to it [17:09:52] and then sort out whether it is or isn't the case [17:10:21] QueenOfFrance: done [17:10:56] liangent: :) better safe then sorry :) [17:13:54] QueenOfFrance, kinda pointless now [17:15:12] Krenair: well, I have no idea what bug is, etc, but it's always good to limit the visiblity even if it's been seen already [17:15:15] It's already been posted on bugzilla publicly, which means that it went to wikibugs-l, hundreds of mailboxes and public list archive sites [17:15:28] Yeah but still [17:15:47] I dunno, maybe you're right and it's pointless but :) [17:17:24] csteipp, ^ [17:23:11] oh, right, the email [17:23:44] Reedy: might want to notify the user [17:27:52] Uh [17:28:01] I think we might have to lock the user until we get in touch with him [17:28:09] to prevent others taking over the account in the meantime [17:28:19] Yeah, if someone who speaks zh can notify the user, that needs to happen. [17:28:31] I don't, but I will lock in the meantime [17:30:00] can't lock it [17:30:05] [39c98376] 2013-05-17 17:29:50: Fatal exception of type MWException [17:30:07] csteipp: ^ [17:30:11] blurg [17:30:39] anyway I tried to reach that user via email and gtalk. though gtalk offline [17:33:30] greg-g: more interesting is the error I got on CA on testwiki [17:37:31] QueenOfFrance: ? [17:37:36] QueenOfFrance: It's not any more interesting as it's effectively the same error [17:37:37] Exception from line 1766 of /usr/local/apache/common-local/php-1.22wmf4/extensions/CentralAuth/CentralAuthUser.php: Bad user row looking up local user 維基小霸王@zhwikiquote [17:37:47] DON'T PASTE!... oh, good ;) [17:37:50] Reedy: well, more interesting than the one I got from meta I mean :) [17:37:54] There's no password in it [17:38:03] #1 /usr/local/apache/common-local/php-1.22wmf4/includes/SpecialPage.php(628): SpecialCentralAuth->execute('???????????????') [17:38:25] If we were exposing passwords in errors from other people that'd be even more of an issue [17:39:11] QueenOfFrance: I'll just null the users password [17:39:20] Reedy: that means they can't log in tho [17:39:21] Reedy: QueenOfFrance, liangent: the global account is locked [17:39:27] pgehres: thank you [17:40:07] Reedy: if Liangent is able to contact him thru gchat, wouldn't it be better to leave it locked, unlock it when he shows up, and so on? [17:40:17] Can do, yeah [17:41:13] New password googlenotgone [17:41:28] lol [17:41:39] <^demon> My password is "password." [17:41:42] <^demon> I never forget it. [17:41:50] And when you do, we can remind you [17:42:00] <^demon> Hey Reedy, what's my bank password? [17:42:07] hm, can't even see that user's special:centralauth page... [17:42:11] Password [17:42:15] <^demon> Thanks! [17:42:17] Just to keep them different [17:42:46] wait, it's not ^password [17:42:47] :o [17:43:23] That's hard to type [17:43:36] <^demon> So, my username has actually broken a fair number of software. [17:43:42] <^demon> People don't escape regexes properly ;-) [17:44:13] <^demon> MW has always worked fine, until ^ was added to the username blacklist on enwiki so I couldn't register it now if I wanted. [17:44:14] Reedy, why are passwords visible like that in the first place? [17:44:27] Because PHP doesn't know it's a password [17:44:28] <^demon> Stacktraces leak all parameters. [17:44:32] How would/could it? [17:44:38] <^demon> I complained awhile back about this, but there's no non-crap way to fix it. [17:45:47] That's why errors stopped getting sent to users wasn't it? They might accidentally post their password. Shell users should be less likely to do this... [17:46:04] so was that line from when the user typed their password in? [17:46:38] It was from a request where the user had to send their password to the servers, yes [17:47:08] The passwords are not usually stored unencrypted - unless they hit errors like in this case [17:47:30] Yeah, this was specialuserlogin [17:48:05] Could maybe pass the password in an object or something? Crap way to fix it though... [17:48:51] Yeah.. bug 30714 in case anyone wants to work on it :) [17:49:39] Reedy: were you able to duplicate the db call in eval.php, that is failing? Apparently fenari doesn't like my key atm [17:49:55] I didn't try [17:50:20] let's see [17:51:13] 維基小霸王@zhwikiquote [17:51:47] Or just try to select on zhwiki where 'user_name' => '維基小霸王' [17:52:06] Yeah, that's what I was going to do [17:52:13] pgehres wasn't able to paste it into the eval shell [17:52:22] mysql:wikiadmin@db1019 [zhwikiquote]> select * from user where user_name = '維基小霸王'; [17:52:22] Empty set (0.02 sec) [17:52:28] Oh, zhwiki? [17:52:32] either [17:52:44] global user id 15195490 [17:52:45] There's a row on zhwiki, not on zhwikiquote [17:52:53] zhwiki user id 1010813 [17:53:55] pgehres: can you see if they have a localuser entry on centralauth for zhwiki? [17:54:04] Actually, that shouldn't matter [17:54:09] http://toolserver.org/~quentinv57/sulinfo/%E7%B6%AD%E5%9F%BA%E5%B0%8F%E9%9C%B8%E7%8E%8B [17:54:12] Yes, this is zhwiki not zhwikiquote according to the full stack trace you posted [17:55:15] Well, the "username" was @wikiquote [17:55:29] suggesting a remote lookup [17:56:20] oh. hm. [17:56:31] There is a localuser row on zhwiki [18:02:28] csteipp: pgehres: We should just tell the user to login to zhwikiquote ;) [18:03:45] Reedy: and we can do it for that user because we've already known the password [18:04:48] Okay, we found the issue [18:04:54] centralauth.localnames is corrupt [18:05:09] ? [18:05:28] liangent: Interesting to have a username in chinese but an english password [18:05:39] queryunattached was trying to load a user identified in localnames that had no local account [18:06:32] <^demon> Krenair: Yeah that was part of it. And while we send appropriate headers telling stuff to not cache, who knows what a misbehaving proxy somewhere in between might do. [18:08:46] Reedy: that's an extremely common case. we can't input a chinese password without copy-paste [18:08:59] Ahh [18:10:42] Reedy: it seems many developers here are not aware of the scenario of working with cjk [18:12:11] Not suprising [18:15:10] So pgehres is running a query to see if we have too many other entries in localnames that don't actually have a local wiki account. If there are lots, we'll have to run something to clean that table up. If it's a few, we'll just do it manually. [18:16:06] No idea how that localname entry got in though-- maybe a failed rename? Or aborted login.. [18:26:22] I can't find any way to modify exception stack traces :/ [18:26:51] <^demon> Yeah, we've got a bug about it. [18:27:07] What you mean https://bugzilla.wikimedia.org/show_bug.cgi?id=30714 ? [18:27:16] <^demon> Yup. [18:27:27] <^demon> Summary could be a bit wider, it's certainly not constrained to that codepath [18:27:35] Yeah that's the one I was trying to solve... [18:28:05] I can catch all exceptions and look through stuff like function/argument details, but I don't think I can modify that information to redact stuff [18:28:37] stack trace to a string [18:28:44] regex replace known function calls [18:28:57] can you say HACK? [18:29:38] <^demon> In tons of languages, if we put it in an i18n file. [18:48:18] Reedy, I was hoping to be able to do it without getTraceAsString :/ [18:48:39] (getTrace provides this info in a structured way) [19:03:29] IRC is down ? [19:03:42] then how would you be talking on it [19:04:03] Frakir: the wikimedia IRC server? [19:04:22] it's back [19:04:23] Frakir: potentially http://lists.wikimedia.org/pipermail/wikitech-l/2013-May/069459.html [19:04:26] we just rebooted the server [19:04:29] Frakir: [19:04:36] Ok [19:28:42] /join #en.wikipedia => No such channel [19:29:27] Frakir: it's #wikipedia-en [19:29:34] Frakir: you mean on irc wikimedia? [19:29:41] it´s down atm [19:30:10] irc wikimedia yes, I can connect but not join channels [19:31:30] same here [19:31:42] you can connect to the server [19:31:42] It's being worked on [19:31:48] ^ [19:32:05] was sbout to say that... but yeah.. [19:42:34] so the service was restarted when the system was rebooted for a kernel upgrade [19:42:48] which has the side effect of clearing all the channels [19:43:03] daemon is running again, but channels arent being recreated [19:43:13] * RobH isnt point on this, just relaying info to you guys ;] [19:54:08] Ok folks irc should be ok now [19:54:12] feeds are running [19:54:54] it's back [21:02:08] i uploaded a new version for this image, a less saturated in color, but the thumbnail on the wikipedia page it's still saturated: http://commons.wikimedia.org/wiki/File:Montaje_Antofagasta_II.jpg [21:02:59] nevermind [21:03:34] :) [21:03:42] anyway, it really took some time to update the thumbnail, the big one that appears on the wiki page [21:03:46] why? [21:03:57] i cleared cache and pressed CTRL+F5 [21:04:00] is it a bug? [21:04:42] it took more than an hour to update [21:31:15] Reedy, sorry about 48589. I think 'edits' => 1, 'editComments' => 1, 'spacing' => 1, 'benchmarks' => 1, 'email' => false should be fine [21:31:30] Krenair: It was only fluke that I actually noticed it [21:32:07] I really hate all the autopromotion stuff [21:32:45] heh [22:19:47] gn8 folks