[02:48:23] Tech team, are we aware of the issues with the Create Account page? There are multiple OTRS tickets in queue complaining that it's blank. [03:00:53] Matthew_: probably not because its in otrs, I recommend filing a bug and poking whomever is around ( TimStarling maybe) [03:02:33] That would make sense. I was seeing if it was a known issue before I filed a bug in BZ. [03:03:31] duplicates in bugzilla are easily handled, not having a report at all is even worse [03:04:38] Fair enough. [03:13:13] p858snake|l: https://bugzilla.wikimedia.org/show_bug.cgi?id=44298 enough? [03:13:47] most likely, unless any of those tickets have more techinical info in them that can be shared [03:15:54] Hmmm... I don't see any right off the bat (most of the users were very ... vague). [08:04:02] Couldn't find a bug about it: https://commons.wikimedia.org/wiki/Commons:Village_pump#.22Error_generating_thumbnail_.E2.80.93_The_source_file_for_the_specified_thumbnail_does_not_exist..22 [12:31:56] hello, what is the problem with : https://commons.wikimedia.org/wiki/File:Halfanim.jpg ? [12:56:40] matanya, hmm, more thumbnail issues. And that one was even uploaded before the Datacenter move... [13:09:22] matanya: check the village pump, there are several reports [13:09:32] yep, same error [13:10:03] I didn't find a bug for it this morning, maybe it's something mark would fix in 15 as usual [13:10:07] *15 s [13:26:22] I should write a summary email later today... [14:23:35] thanks Nemo_bis [17:36:50] spagewmf: ori-l: there is an unsynced change from matthew flaschen in php-1.21wmf7 and i need to scap [17:36:55] it looks like it should be safe but was hoping to confirm [17:37:01] commit 64800ca321984abad97a80b09a4cd4b91fd4f0cc [17:37:01] Author: Matthew Flaschen [17:37:01] Date: Wed Jan 23 23:21:08 2013 -0500 [17:37:02] (bug 44298) EventLogging hotfix for IE [17:37:03] [17:37:04] Change-Id: I2f4d09d04f6e4f010a39c28b3bfe2fc41e378614 [17:37:20] i have folks waiting on me, so im going to sync regardless in a minute [17:37:49] actually, i'll probably revert the change if i don't hear differently [17:40:37] awjr, related, don't know if you've seen: https://bugzilla.wikimedia.org/show_bug.cgi?id=44298#c8 [17:41:34] Krenair: thanks i saw that; i didnt feel comfortable deploying that particularly since matt said 'when tim gives the go ahead' [17:41:40] so i reverted it [19:37:28] * AaronSchulz sighs at http://upload.wikimedia.org/wikipedia/commons/thumb/e/eb/John_Angelos.JPG/160px-John_Angelos.JPG [19:39:21] oh noes - does source file actually exist ? [19:40:44] yes, there is some normalization going on that breaks the existence check in the db [19:40:57] LeslieCarr: http://upload.wikimedia.org/wikipedia/commons/e/eb/John_Angelos.JPG [19:41:09] well at least that picture fits on my screen... [19:47:01] AaronSchulz: was this bug filed? it was first reported yesterday on a couple village pumps [19:47:14] no idea [19:47:17] and is http://www.wecowi.de/images/thumb/6/66/Yosemite_Half_Dome.jpg/120px-Yosemite_Half_Dome.jpg the same? [19:48:00] probably not [19:48:35] AaronSchulz: anything wrong on the swift side? [19:48:39] or anything I can help you with in general? [19:50:31] this is just the DB part afaik [19:51:59] AaronSchulz: you know we're still using pmtpa's imagescalers, right? [19:52:34] I know [19:52:42] k [19:53:27] $file = wfLocalFile( 'John_Angelos.JPG' ); [19:53:28] var_dump( wfGetDB( DB_SLAVE )->selectRow( 'image', '*', array( 'img_name' => $file->getName() ) ) ); [19:53:30] var_dump( $file->exists() ); [19:53:40] paravoid: so the first dump shows the row, the second is false :s [19:54:25] haha [19:54:28] $file->loadFromDB(); [19:54:30] var_dump( $file->exists() ); [19:54:34] not its true [19:54:43] wtf, memcached? I thought negatives were not cached [19:55:34] $file->loadFromCache(); [19:55:36] var_dump( $file->exists() ); [19:55:37] false again [19:55:43] meanwhile, i'm having "fun" with ceph [19:56:42] huh, it does indeed cache negatives [19:57:44] filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=44319 [20:21:37] paravoid: you are sure we are still using pmtpa for scaling? It would explain this problem [20:21:57] binasher: which conf was that that implied otherwise? [20:22:20] root@ms-fe1:~# grep thumbhost /etc/swift/proxy-server.conf [20:22:20] thumbhost = rendering.svc.pmtpa.wmnet [20:22:22] So, today I notice (a) Oh, hey! I'm in CREDITS now. (b) ... with my name spelt wrong. :-) [20:23:13] paravoid: anyway, I think the problem is with not using one memcached cluster [20:23:28] AaronSchulz: which actual hostnames matter for this? it would be commons.wikimedia.org, etc., not upload.wiki? [20:24:35] binasher: I thought it would be upload.wikimedia [20:25:57] AaronSchulz: but re: UW, don't uploads go via an actual project url? [20:26:21] oh, yes the actual uploads do [20:26:35] the text squids are mapping thumbs to rendering.svc.eqiad [20:26:36] both api and the special page would be commons.wikimedia.org [20:26:41] swift howerver is mapping rewrite_thumb_server => "rendering.svc.pmtpa.wmnet" [20:26:58] so that probably adds to the memcache inconsistencies [20:27:33] ugh [20:27:56] binasher: text squids mapping which thumbs to rendering.svc.eqiad? [20:29:08] squid.conf.php:acl thumb_php urlpath_regex ^/w/thumb(_handler)?\.php + text-settings.php: '=thumb_php' => 'rendering.svc.eqiad.wmnet', [20:29:18] and then we have [20:29:29] squid.conf.php:acl swift_thumbs url_regex ^http://upload\.wikimedia\.org(/+)[^/][^/]*/[^/][^/]*/thumb/ + upload-settings.php: '=swift_thumbs' => 'ms-fe.svc.pmtpa.wmnet' [20:29:43] where swift is then using rendering.svc.pmtpa [20:29:52] what would link to thumb.php/thumb_handler.php besides random URLs on the internet? [20:29:58] is UW linking to that directly? [20:30:14] paravoid: non-public wikis [20:30:21] iirc [20:30:23] thumb.php is used for wikisource in some places [20:30:32] ok, both true [20:30:34] and UW does an internal curl [20:30:45] not UW, but upload stash [20:30:47] but that doesn't explain the commons issue above [20:31:07] AaronSchulz: is there any other case where UW or anything else accessed via commons.wikimedia.org and thus hitting an eqiad apache would need to access filestat memcached keys? [20:32:57] not that I can think of [20:33:50] heh, well there is parsing [20:33:55] * AaronSchulz duhs [20:34:04] hmm? [20:34:38] :P [20:34:52] binasher: an eqiad apache parsing a page may call wfFindFile() a bunch of times [20:36:40] yeah [20:36:56] we need to use the eqiad scalers universally [20:36:59] alas [20:37:43] yep [20:38:29] paravoid: can you make the changes? :) [20:39:02] paravoid: would you be down to do do that now? i'm not sure what if anything is needed to make a swift rewrite_thumb_server puppet change live [20:40:47] sec, I'm in the middle of a ceph debugging session [20:43:12] changing is easy enough [20:43:19] a puppet change + restart of the proxies [20:43:44] but it wouldn't be very nice from a performance pov [20:44:24] I'll do it in a few minutes [20:44:27] having the pmtpa image scalers use eqiad memcached probably won't be any better ;) [20:44:49] haha, I guess not :) [21:01:05] binasher, AaronSchulz: done btw [21:02:15] huzzah, a new day of higher latency is finally here! [21:03:13] it might just be my internet. but is anyone having difficulty getting to the wiki [21:03:34] !log Synchronized payments cluster to fe035f6b5a55946b729f [21:03:45] Logged the message, Master [21:05:04] Seddon: works reasonably swiftly for me. [21:05:16] Seddon: can you send me a traceroute with your ip ? [21:07:12] Actually dont worry, it seems to be a problem my end. struggling with everything.... irc seems to be fine though [21:16:10] "irc seems to be fine though" ... "Ping timeout: 248 seconds". Famous last words? [21:20:02] binasher: those thumbs are working now \o/ [21:20:59] maybe I won't need to fudge the key [21:21:40] AaronSchulz: :) [21:22:09] AaronSchulz: what changed? [21:23:53] paravoid: no SAL entry? [21:24:15] there was one, but too convoluted I guess [21:24:18] 20:56 paravoid: gradual restart of ms-fe[1-4] proxies [21:24:29] you mean "cryptic"? [21:26:05] :) [21:28:21] paravoid: I updated your entry, heh [21:28:36] thanks [21:28:41] robla: swift 404 handling goes to eqiad now [21:28:50] aha, thanks! [21:51:57] is the process for running sync-dir same as before? [21:53:14] ori-l: was that you approving changes that add count(*) on watchlist queries? [21:53:22] domas: no [21:54:35] First I hear about this. Which changes? You're probably confusing me with someone else.. [21:56:03] ori-l: Yeah, it is [21:56:13] Reedy: thanks. [22:03:59] domas: and GROUP BY and HAVING? [22:04:29] ori-l: maybe some other ori [22:04:36] aaronschulz: yes [22:04:52] on the upside, wikipedia editing thing is tiny project [22:04:55] so you can do whatever you want [22:05:13] domas: does your IRC client normalize names to lowercase when you tab-complete? [22:05:36] domas: Sorry, I'm a bit unnerved by the humor. There isn't another Ori, to my knowledge. Do you have the SHAs or change-ids for the commits? [22:06:58] aaronschulz: I never use tab completion [22:07:07] hardcore [22:07:37] probably this one - https://gerrit.wikimedia.org/r/#/c/27134/ [22:07:38] AaronSchulz: he also never typos [22:07:48] AaronSchulz: his fingers are the perfect typing machines [22:07:52] From bugzilla " Why do IRC clients always pick the wrong name to autocomplete to? siebrand: they always pick right for me!" [22:07:53] ;) [22:08:05] reedy: yeh [22:08:10] Did you stop then? [22:09:57] domas: Can you look again? I did not introduce or modify queries in that change set. [22:10:04] you did not [22:10:09] it just started using code that does those queries [22:10:19] I was too lazy to search who introduced it [22:10:25] this is the general loop of greatness: [22:10:35] a) introduce shitty code "because nobody will use it anywhere, so it can be there" [22:10:43] b) start using it everywhere "it is already here" [22:10:43] :) [22:11:19] ori-l: I saw that when looking at this - https://gerrit.wikimedia.org/r/#/c/45366/ [22:12:02] count(*) backed by memcached query is my favorite anti-pattern [22:12:50] I don't see memcached in that patch [22:15:57] aaronschulz: it is in the page counts code path [22:17:06] InfoAction::pageCounts() and InfoAction::pageInfo() [22:20:09] domas: it just started using code that does those queries [22:20:12] also not true [22:20:43] * AaronSchulz looks at http://en.wikipedia.org/wiki/Wikipedia?action=info [22:22:11] is it also counting revision entries? [22:22:35] no; $pageCounts is already populated [22:22:51] it does not cause the problematic code to be called more often than it would be otherwise. [22:23:00] haha, I'm currently melting the cluster with this on ANI [22:23:06] ~35 sec to load [22:23:17] MaxSem: good page choice [22:23:18] Number of page watchers 5,697 [22:23:26] thanks to lack of proper threading system [22:24:04] * MaxSem thows flowers on LQT's grave [22:25:19] MaxSem: now http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incidents?action=info is fast until someone edits it again [22:25:25] which will be a few seconds probably ;) [22:25:56] Total number of edits 677,669 [22:25:57] domas: good stuff [22:26:07] domas: anyways, I'm only pushing back because a lolwat from you is pretty damning, and someone reading the channel log casually might still think that I was responsible. But w/e. [22:26:17] of course it scanned it again doing a big DISTINCT query [22:26:45] at least those are index scans [22:29:48] aaronschulz: I think it all has to be migrated to flash-backed databases [22:29:55] using spinning rust is not good for modern web environments! [22:30:02] people should allow to write whatever queries they want [22:30:19] ori-l: nah, I mentioned you just because I saw you somewhere [22:30:25] not because I feel like I should unload on you [22:30:27] :) [22:30:44] ok!:) [22:30:49] so, domas is still domas [22:30:49] also [22:30:50] "[13:53:14] ori-l: was that you approving changes that add count(*) on watchlist queries? " [22:30:54] domas, alternatively, we can switch to postgres [22:30:57] yup [22:31:01] magic [22:31:02] instagram I hear uses postgres [22:31:12] I'm already working on manifests [22:31:50] isn't wikipedia going to use mariadb [22:31:55] it has much better optimizer I hear!!!!! [22:32:11] * AaronSchulz hands domas more vodka [22:32:33] he drinks and raves about maria [22:32:36] must be a gf back home [22:33:21] naw, maria is just monty's alter-ego [22:34:31] likes not getting an Oracle copyright message on console [22:34:48] I have some 18-y-o whisky in a glass next to me [22:34:55] okay, we've scared Asher off [22:35:35] mm, what kind ? [22:35:50] sorry if my joke was in poor taste [22:37:10] speaking of which, can someone review https://gerrit.wikimedia.org/r/#/c/36155/ ? [22:41:26] lesliecarr: jamesonm [22:41:35] had glenmorange yesterday [22:56:55] didn't postgres add indexed count(*) in like the previous release or so? [22:56:57] maybe 9.x? [22:57:47] and started making coffee [22:59:32] http://wiki.postgresql.org/wiki/Slow_Counting [22:59:35] there [22:59:39] Note that the following article only applies to versions of PostgreSQL prior to 9.2. Index-only scans are now implemented. [23:00:34] meh [23:00:48] domas: ^^^ [23:01:35] you obviously read it wrong and are talking about different thing at all [23:01:38] \o/ [23:01:53] which I expected from early on, by writing "and started making coffee" comment :) [23:02:12] now PG is able to do index reads the way MySQL does them [23:02:17] it was major problem for very long time [23:02:19] yep [23:02:28] yes? [23:02:59] didn't say it's something that mysql doesn't do [23:03:06] this isn't "indexed count(*)", this is "count(*) that reads index-only" [23:03:19] indexed lookup and index scan is somewhat different [23:03:40] ok [23:04:01] now, the way it was said [23:04:08] someone mentioned postgres before, I was just thinking how awesome would it be to have that while running on postgres < 9.2 :) [23:04:11] you obviously didn't mean that this is something where PG is more like MySQL [23:04:52] see above [23:05:16] I remember this being a huge problem on postgres and I remember reading lately how it got fixed [23:05:17] hehe [23:05:28] guys, are you arguing over who's being more sarcastic?:P [23:05:31] well, technically having to crawl an index is still a huge problem! [23:05:43] all 670k rows! [23:05:52] for WP:ANI or whatever [23:15:11] ?action=info has a caching layer, in theory, I think. [23:15:22] There's memcached support in there somewhere. [23:15:50] I don't think it's really much worse than the dozens of other ways you can cause a huge table scan. [23:16:50] Susan, the point is that even cache updates are brutally slow [23:18:18] Information wants to be free. [23:19:36] WP:ANI only has 10 edits, by the way. ;-) [23:20:20] https://en.wikipedia.org/w/index.php?title=Wikipedia:ANI&oldid=17814003 # Heh, I do this every once in a while. [23:30:55] Susan: do agree on this being lowest? https://bugzilla.wikimedia.org/show_bug.cgi?id=44311 [23:31:40] ah it wasn't that oen [23:31:54] I don't use undo very often. [23:33:10] That's a n00b-benefit request. [23:43:41] gn8 folks