[00:01:18] New patchset: Asher; "frontend cache tier udplog data for graphite" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2523 [00:01:41] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2523 [00:01:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2523 [00:01:42] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2523 [00:01:43] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2523 [00:04:27] New patchset: Asher; "more capable and efficient version of sqstat" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2524 [00:04:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2524 [00:04:55] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2524 [00:04:56] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2524 [00:09:48] New patchset: Asher; "add path to proc name for silly nagios monitoring" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2525 [00:10:10] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2525 [00:10:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2525 [00:10:13] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2525 [00:10:13] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2525 [01:59:37] gn8 folks [02:19:18] !log LocalisationUpdate completed (1.18) at Sun Feb 12 02:19:17 UTC 2012 [02:19:23] Logged the message, Master [05:14:09] New patchset: Asher; "much more accurate at counting pageviews and edit submission" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2526 [05:15:34] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2526 [05:15:34] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2526 [05:43:55] nighty [05:54:33] New patchset: Asher; "fix edit tp99 reporting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2527 [05:54:54] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2527 [05:54:54] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2527 [05:54:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2527 [06:21:42] New patchset: Asher; "variable scope fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2528 [06:22:09] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2528 [06:22:10] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2528 [10:03:37] New patchset: Mark Bergsma; "Give the API apaches a higher MaxClients setting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2529 [10:04:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2529 [10:04:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2529 [10:04:12] New patchset: Dzahn; "let the LVS HTTP check on api cluster check a real api.php URL instead of Main_Page" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2530 [10:27:08] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2530 [10:27:09] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2530 [12:00:08] I'm getting exclusively blank/broken results from ENWP's search suggestions api. hmm... [12:02:26] there are two java processes on search8, can someone please kill both of them and do "/etc/init.d/lsearchd start"? [12:02:32] YairRand, thanks for report [12:07:43] rainman-sr: ok [12:08:00] rainman-sr: so this indexing job that runs on sunday mornings... [12:08:03] has problems every week [12:08:16] besides trying to fix it, can we move that to another day when it doesn't wake us up ever sunday morning? ;) [12:09:19] !log Killed lsearchd processes on search8, restarted [12:09:21] Logged the message, Master [12:09:31] it is logrotate [12:09:48] logrotate rotates the indexes? [12:09:55] no, the logs [12:10:00] in /a/search/log [12:10:07] that cuases the problems [12:10:08] ? [12:10:33] apparently what happened now is that /etc/init.d/lsearch stop didn't work, i.e. the process was still up [12:10:39] yes that's this problem [12:10:42] another problem is the rsync [12:11:06] we think it's because the rsync fills the linux caches and pushes out the old index being served [12:11:08] does that make sense? [12:11:15] so maybe there should another layer or robustness of doing kill -9 or smth if stop doesn't work [12:11:25] yeah [12:11:34] we want to change the setup quite a bit [12:11:37] mark, yep, that is a problem, that's why rsync runs with limited bandwidth [12:11:44] yeah but it doesn't work sufficiently [12:11:44] but at some point the indexes just get too large [12:11:53] the current configuration was set up some 2 years ago [12:12:05] asher wants to replace rsync with one that does a posix fadvise where it doesn't pollute the caches [12:12:09] yes I know [12:12:11] we're gonna fix it now [12:12:21] having many more people helps ;) [12:12:23] well oki, that sounds good [12:12:40] although having rsync polluting the cache is kinda part of index warmup [12:12:43] we want to hire a search developer as well [12:12:50] true [12:13:00] at some point there these pages will need to cached by linux [12:13:07] yes [12:13:11] having boxes with more memory would also help [12:13:13] and/or ssds [12:13:16] yep [12:13:19] we can do that with the new eqiad cluster perhaps [12:13:32] yes, that would be very helpful [12:13:37] anyway, any help you can give Peter in answering questions about the setup would help a great deal [12:13:44] we know you're busy and don't have time for this really [12:13:58] but the sooner we can get it fixed, the sooner we can leave you alone completely ;-) [12:14:09] if he asks i answer. sometimes e-mails get lost, so if i don't reply immediately, just poke me agone [12:14:15] alright [12:14:16] thanks [12:14:21] np [12:14:55] anyway [12:15:09] we can probably rotate the days for this indexing, right? [12:15:14] search dies every sunday now [12:15:22] and it would be awesome if that could be on a normal work day instead ;-) [12:16:13] well, the rationale for sunday is that the load is then the smallest, and thus restarting the search deamon has the smallest effect [12:16:32] in that respect, saturday would be best [12:16:36] however, friday is also fairly low load [12:16:51] and then we are actually available to fix issues [12:17:48] well ok as long as the load is small it think there is no problem in moving it [12:18:02] ok [12:18:04] /etc/logrotate.d/wikimedia-task-search [12:18:06] this is the file [12:18:11] thanks [12:18:11] which is probably know [12:18:14] *you [12:18:22] yep [12:18:31] we'll probably gradually move your scripts/cron jobs/whatever into puppet [12:18:49] yep that would also be good [12:18:51] if you still want to be involved, you can easily put changes into gerrit/git [12:18:54] if you don't, that's fine too [12:19:39] well i think it would be useful if i still had access in case something breaks really hard, but i would prefer to delegate any further development and maintenance to someone new [12:19:48] yes [12:21:00] hmm [12:21:05] we have some boxes with 192 GB memory now too [12:21:10] I wonder how search would work on those [12:22:52] ah, well it would work really well i think :) [12:23:14] hehe [12:23:33] we'll see how it goes with the new eqiad servers as well [12:23:43] there should be no problem in running the eqiad servers concurrently with the pmtpa ones, right? [12:24:14] basically it's one network, it's just that they're ~30ms away [12:25:51] well, not sure. i think it would be good to make sure that when pmtpa servers do distributed search on the backend, they only look for pmtpa servers [12:26:11] why do you think so? [12:26:20] do the search servers talk to eachother? [12:26:54] yep, indexes are split and distributed around the cluster, so when a search query arrives, the box needs to decide who to contact to perform the query [12:27:23] but are there multiple queries following then, or just one? [12:27:37] for instance, to do a single en.wp search one would typically recieve a query on search1, use highlighting on search14 or smth, and spell checking another host [12:27:45] ok [12:28:14] but if both data centers can stay within their own data center except for the first search query from mediawiki, then it should be fine, right? [12:28:15] it is all via RMI (which is bad for load balancing but has good connection maintenance) [12:28:26] yep, right [12:28:31] ok [12:28:47] we're probably going to have apaches active in only one data center in the near future [12:28:54] but we would prefer to have search active in both, if possible [12:29:14] and of course we should make sure that one data center can run on its own if necessary [12:29:23] so all indexes/clusters should exist in both data centers [12:29:30] well maybe it woudn't be much of a problem that they are 30ms apart, we would have to test on some machines [12:29:32] rainman-sr: ohi! care to come to berlin on june 1-3? [12:29:49] 30ms is a problem if there are many queries in succession [12:29:58] for example with mysql queries, there are often many to build a single wiki page [12:30:06] and adding up 30ms many times, becomes a problem [12:30:12] mark, yep, well in that case it would be really good to have two mutually exclusive clusters, so that only mediawiki knows there are two of them, but they are not aware of each other [12:30:12] but we were hoping this wouldn't be so much a problem with search [12:30:22] rainman-sr: yeah that is the plan [12:30:52] Daniel_WMDE, hi Daniel! I think I will be very busy then, sorry :( I think it should be better next year :) [12:31:24] rainman-sr: what would it take to not have to split the indexes btw? [12:31:32] what is the size of the largest index right now, approx? [12:32:07] lemme me check [12:32:26] rainman-sr: i'd love an opportunity to talk to you about integrating fast category traversal with the search backend. i have a prototype running on the toolserver now (currently only simplewiki though) [12:32:53] rainman-sr: don't want to keep you now, just want to plant this in the back of your head :) [12:34:32] mark, for en.wp it's around 80gb [12:34:37] ah ok [12:34:44] we could do that in memory if we wanted to [12:34:49] that should be feasible [12:35:56] my estimate is that it's going to double in 2 years [12:36:06] memory prices will halve in 2 years [12:36:09] so I think that's fine [12:36:40] Daniel_WMDE, oki :) [12:36:43] (index for fast category traversal on enwp: 0.5gb. just saying) [12:37:42] mark: there's also something (totally unrelated) i wanted to ask you... i recently came across zeromq. do you know it? could that be useful to us? it sure sounds cool... [12:38:18] well oki guys, i need to go help out with lunch :) talk to you later [12:38:20] yes we use it in fundraising [12:38:27] enjoy rainman [12:38:31] and thanks for your help/input [12:46:20] guillom: Don't know if you're the right person to mention this to, but I just loaded https://blog.wikimedia.org/2012/02/11/mediawiki-1-19-deployment/ on my laptop and got the mobile version(?) [12:49:48] Jarry1250, I've had similar reports in the past, but never any indication as to what might be causing this. [12:55:23] Yeah, can't even recreate myself. [14:53:32] is there a way to make pdfs appear sideways? [14:53:58] my upload of landscape pages are treated like portrait [18:51:57] Reedy, around? [18:53:04] Or anybody else. [18:53:04] For some reason, the API seems to be returning a long since expired block from 2006 for http://en.wikipedia.org/w/api.php?action=query&list=users&ususers=Mistress%20Selina%20Kyle&usprop=blockinfo [19:23:24] How can an infinite block expire? [19:23:53] Simply by definition it can't [19:26:53] not expired, unblocked [21:20:29] zzz =_= [22:09:03] Did the CSS on Wikipedia recently change? Everything's bold for me [22:09:36] matthewrbowker, did you try clearing your cache' [22:10:12] Nemo_bis: No luck... [22:10:44] * Nemo_bis just asks the silly easy questions to be sure [22:11:11] It's no problem, sometimes the biggest problems take the simplest fixes [22:20:30] Anyone? I've had no luck with it, so I'm curious. I'm also seeing it on -tech wiki. [22:32:55] matthewrbowker: try using a different browser [22:33:55] TimStarling: OK, I'll be right back. Qiting Firefox. [22:47:10] TimStarling: Wow, that took longer than normal, IE hates me just like I hate it. Anyway, it displays normal in IE, it doesn't display normal in the production Firefox. [22:48:12] it looks fine for me in firefox [22:48:25] which is expected because it's just you in here complaining, not a million people [22:48:44] maybe you should try with a new profile [22:49:06] that would tell you if it's a configuration problem [22:49:48] or grab portable firefox and test with that [22:50:29] OK. 'sec, I've got to quit this time. [22:52:52] TimStarling, p858snake|l: No luck :( I'm running a new profile on Nightly (Firefox's nightly build) [22:53:24] is that the same version as you were using before? [22:53:57] Wait... No. Darn it. [22:55:12] Same result with a new profile and with the old (and yes, they're the same browser this time) [22:56:30] so what version were you using at first? [22:56:56] 10.0 Now I'm using 13.0 with the same result. [22:57:53] maybe you should try irc.mozilla.org [22:57:59] it doesn't sound like it's our fault [22:58:30] 13.0? [22:58:56] TimStarling: I asked here cuz it was only Wikipedia. But I can ask there. [22:59:01] DabPunkt: Nightly build [23:22:07] gn8 folks