[00:00:03] (I still want those +/- for my watchlist [00:00:04] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [00:00:05] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [00:00:12] table.mw-contributions-table .mw-plusminus-pos { display: none } or something like that [00:00:20] ok lets try that [00:00:20] .http://meta.wikimedia.org/wiki/Special:Contributions/Okeyes_%28WMF%29 [00:00:25] huh? [00:00:31] ah, clipboard [00:00:44] ok so that works [00:00:57] thanks folks for the tips [00:01:05] but I have a minor quibble [00:01:06] ". . (+834)‎ . . " [00:01:14] is there a way to get rid of the periods [00:01:25] (I want to manipulate this back to before) [00:03:17] not easily [00:03:18] table.mw-contributions-table .mw-plusminus-pos:before { content: "" } maybe? [00:03:31] and table.mw-contributions-table .mw-plusminus-pos:after { content: "" } [00:03:37] just guessing [00:03:41] ok I'll try that [00:03:45] that won't work [00:03:55] ok then how should i do it [00:04:20] ah no [00:04:24] I'm not worried about the length of any code, I just care it works [00:04:26] :before works inside [00:04:53] right [00:04:59] what do you mean, sorry I'm not very techy [00:05:07] New patchset: Lcarr; "requiring facter files before running tcp tweaks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2630 [00:05:26] Guest50878: i'm afraid the only way would be by hacking it using javascript replace [00:05:41] I wouldn't mind that [00:05:43] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2630 [00:05:44] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2630 [00:05:48] can you provide some code [00:05:51] Danny_B|backup: maybe something with non-box element would do w/:before ? no idea [00:07:06] Guest50878: i'd rather file a bug requesting some markup for that so it's easily revertable to previous look via css [00:07:46] well if its posible with css or js, i'm all ears [00:09:20] strange

no class [00:09:47] I don't know what that means, but it doesn't sound good [00:10:46] whatever, dots are fine for me :) [00:10:48] yeah, that's bug if there is no class then don't output it at all [00:10:58] siebrand: quick question if you're around about https://bugzilla.wikimedia.org/show_bug.cgi?id=33762 [00:11:16] chrismcmahon: i hardly doubt he's around now [00:13:12] Danny_B|backup: same in 1.18wmf1 (r109351) [00:14:05] ok so I'm taking this as a sign that there is no solution [00:14:16] so I guess I'll check with the bug folks [00:14:34] to see if they can install a gadget for reversion to previous contribs views [00:15:05] Guest50878: sure there is some solution, but not within scope of 5-line help on IRC [00:15:11] right [00:15:13] Guest50878: just curious, why do you want the previous version? it looks better, you have scripts that depend on it, some other reason? [00:15:33] no I just don't like the change. it's seems like unnecessary clutter [00:15:42] ok, thanks. [00:15:49] maplebed: some people just like things the way they are used to [00:16:11] maplebed: that's why majority of active users on cswiki switched back to monobook and they hate vector [00:16:18] sure, I just wanted to know (so I wouldn't assume). [00:16:34] * saper just realized it's different :) [00:16:54] well... so there *is* a solution but it's complex. would you be able to give a general overview of whawt the solution would be so i can make further inquires elsewhere or maybe try to research it myself? [00:17:20] well in the old version titles were *mostly* lined up, not the case here [00:17:39] New patchset: Lcarr; "Making initcwnd.erb only generate if puppet fact exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2631 [00:17:55] Guest50878: by far the best long-term solution is to file a bug requesting CSS markup around the whole thing so that you can easily remove it. [00:18:01] Guest50878: find some simple javascript (you will have lots of them in [[MediaWiki:Common.js]] on any larger wiki) that manipulates neighbouring "DOM nodes" [00:18:31] ok thanks folks! [00:18:36] I'll check more into this [00:18:40] bye :) [00:19:03] New review: Ryan Lane; "If is slightly off. See inline comment." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2631 [00:20:28] New patchset: Lcarr; "Making initcwnd.erb only generate if puppet fact exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2631 [00:21:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:24] hey, I just inquired about the diff changes in 1.19 [00:21:30] just a quick question [00:21:44] saper said [00:21:50] something about js manipulates neighbouring "DOM nodes" [00:22:05] can i get a quick example of that so i know what i'm lookingfor [00:22:20] I'm looking at http://en.wikipedia.org/wiki/MediaWiki:Common.js (a big wiki) [00:23:25] I'm no js author but maybe saper can tell me whether http://stackoverflow.com/questions/2558396/jquery-get-neighborhood-element is the right idea. [00:23:35] Is secure.wikimedia.org gone now? [00:24:08] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2631 [00:24:08] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2631 [00:24:33] alot_of_mike: It's down [00:25:00] okay, I thought maybe it was gone for good now that the regular domains support SSL [00:25:04] but not supposed to be [00:25:10] We did intend to keep it up and redirect stuff [00:25:19] beautiful - thanks, Roan [00:26:15] hmm http://stackoverflow.com/questions/2558396/jquery-get-neighborhood-element goes straight over my head [00:27:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.841 seconds [00:28:50] well I have to go pretty soon so thanks for the help [00:31:07] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [00:32:28] PROBLEM - MySQL Slave Delay on db1003 is CRITICAL: CRIT replication delay 193 seconds [00:33:22] PROBLEM - MySQL Slave Delay on db1019 is CRITICAL: CRIT replication delay 246 seconds [00:35:55] RECOVERY - HTTP on singer is OK: HTTP OK - HTTP/1.1 302 Found - 0.001 second response time [00:36:31] RECOVERY - MySQL Slave Delay on db1003 is OK: OK replication delay 0 seconds [00:37:16] RECOVERY - MySQL Slave Delay on db1019 is OK: OK replication delay 0 seconds [00:53:10] PROBLEM - Puppet freshness on ganglia1001 is CRITICAL: Puppet has not run in the last 10 hours [01:02:10] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 7.167 seconds [01:08:19] AaronSchulz: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/111708 [01:08:50] your summary should say "patch by tim starling" ;) [01:09:04] yes it should. can I change it? [01:09:10] meh, too late [01:09:29] * maplebed cotmments instead. [01:10:40] New patchset: Bhartshorne; "porting most recent version from svn - passes along x-forwarded-for and other headers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2633 [01:11:18] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2633 [01:11:19] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2633 [01:13:56] oh noes [01:14:07] my 3 minutes of work will forever go uncredited [01:14:24] * AaronSchulz watches tim rub it in [01:17:27] binasher: can report.py not use e+ notation so easily? [01:17:37] seems kind of annoying overkill [01:18:52] yeah..1.3e+03 heh [01:18:53] seems to just be e+03, 1 sec, so I guess it's OK [01:19:16] but's not exactly that cramped for space [01:19:21] AaronSchulz: the addheaders (for xff) thing crashed. [01:19:24] :( [01:19:29] I think I can fix it though. [01:19:31] in the space of e+03 it could have said the number ;) [01:19:58] totally.. i will change that if it hasn't been done already when i'm back from vacation (i'm taking off in a few min til 2/27) [01:39:54] New patchset: Ottomata; "test commit for git branch push" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2634 [01:39:56] New patchset: Ottomata; "Adding ability to reference observation instances by just trait_sets, without trait values." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2635 [01:39:57] New patchset: Ottomata; "Hacky first work on loader classes." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2475 [01:39:58] New patchset: Ottomata; "Renaming the concept of variables to 'traits'. Allowing trait_sets to be specified so that we don't record HUGE amounts of data." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2477 [01:39:59] New patchset: Ottomata; "base.py - adding schema in comments. Got lots of work to do to make this prettier" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2476 [01:40:01] New patchset: Ottomata; "device_pipeline.py - comments about hackyness" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2479 [01:40:02] New patchset: Ottomata; "Adding loader.py - first hacky loader, just so we can get some data into mysql to work with." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2478 [01:40:03] New patchset: Ottomata; "Buncha mini changes + hackiness to parse a few things. This really needs more work" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2623 [01:40:04] New patchset: Ottomata; "pipeline/user_agent.py - adding comment that this file should not be used" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2480 [01:41:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:46] maplebed: srv226 specieswiki: Could not access `wikipedia-species-local-thumb`; container does not exist. [01:46:10] stafford is very unhappy [01:47:25] what the hell is it doing? http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=load_one&s=by+name&c=Miscellaneous+pmtpa&h=stafford.pmtpa.wmnet&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [01:47:27] AaronSchulz: that's true! it doesn't exist. [01:47:44] ewww: http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&m=&c=Miscellaneous+pmtpa&h=stafford.pmtpa.wmnet&tab=m&vn=&mc=2&z=small&metric_group=ALLGROUPS [01:49:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.028 seconds [01:56:00] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 620s [01:56:27] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 649s [01:59:05] AaronSchulz: robla (and TimStarling): I think I lied before. Looks like the switch to unchunked broke PUTs when it comes from the image generators. [01:59:28] symptom (from the client's pov) - first request for an image fails (empty body); second succeeds and puts it into swift. [02:00:08] I think it's because files generated don't have a content-length [02:00:12] (though they do have an etag) [02:00:12] RECOVERY - mysqld processes on db1035 is OK: PROCS OK: 1 process with command name mysqld [02:00:34] oh, sure [02:00:44] another 1 liner [02:00:47] ok....we just need to add content-length on the generated stuff [02:01:13] http://pastebin.com/7zEevJTZ [02:01:24] (the generated stuff says transfer-encoding: chunked) [02:01:31] !log db1035 is replicating again [02:01:34] Logged the message, Master [02:01:59] what would you think of saying 'only pay attention to content-length if it doesn't have an etag'? [02:02:11] (since, in theory, etag is a better assurance anyways) [02:03:22] maplebed: that's one way of doing it (use chunked for generated, normal PUT for files), but... [02:03:47] ...it sure seems like we should have easy access to content-length [02:03:53] yeah, it's true. [02:03:55] ok, nevermind. [02:04:04] +1 throw content-length in the generated version. [02:04:24] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 5518 seconds [02:05:03] http://www.mediawiki.org/wiki/Special:Code/MediaWiki/111717 [02:05:38] AaronSchulz: do we need to remove the Transfer-Encoding: chunked header? [02:05:47] * robla really hates using string functions for byte counts [02:05:49] or will adding Content-Length magically replace it? [02:05:50] maplebed: did you create the wiki species container [02:05:57] no, not yet. [02:06:00] I'll go do that. [02:06:11] robla: in python? [02:06:26] thumb-handler.php is not explicitely setting a chunked header [02:06:40] jeremyb: see the rev ^ from Aaron [02:07:08] heh [02:07:22] well php i have no idea. python i know ;) [02:07:50] maplebed: oh, and update rewrite.py in svn [02:08:35] AaronSchulz: the species container is made. [02:09:02] * AaronSchulz does some header testing [02:09:40] swift will be a zoological park pretty soon [02:10:40] AaronSchulz: I committed the user-agent / xff header fixes. [02:12:33] no sure where 'chunked' is coming from [02:14:08] AaronSchulz: maybe just automatically added if there's no length? [02:14:41] * AaronSchulz looks at nginx [02:15:02] that diff looks fine to me. [02:15:14] (r111717) [02:16:16] !log on db40: truncating pc008 - 15 [02:16:18] Logged the message, Master [02:17:35] !log LocalisationUpdate completed (1.18) at Fri Feb 17 02:17:35 UTC 2012 [02:17:37] Logged the message, Master [02:17:55] AaronSchulz: am I waiting for you or did you push that out and I should test it? [02:18:06] I didn't push anything [02:18:16] Couldn't if I wanted anyway ;) [02:19:39] maplebed: http://www.ruby-forum.com/topic/188628 [02:21:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:52] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [02:27:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.524 seconds [02:31:33] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Fri Feb 17 02:31:25 UTC 2012 [02:34:32] !log LocalisationUpdate completed (1.19) at Fri Feb 17 02:34:32 UTC 2012 [02:34:33] RECOVERY - Disk space on search1002 is OK: DISK OK [02:34:34] Logged the message, Master [02:34:42] RECOVERY - DPKG on search1002 is OK: All packages OK [02:34:42] RECOVERY - RAID on search1002 is OK: OK: no RAID installed [02:41:27] PROBLEM - Disk space on mw1 is CRITICAL: DISK CRITICAL - free space: /tmp 41 MB (2% inode=87%): [02:42:30] RECOVERY - NTP on search1002 is OK: NTP OK: Offset -0.01181232929 secs [02:44:00] RECOVERY - Disk space on mw1 is OK: DISK OK [02:45:48] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 2s [02:46:27] New patchset: Bhartshorne; "correcting syntax for passing through headers to ms5" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2636 [02:46:33] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:46:50] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2636 [02:46:50] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2636 [02:47:00] RECOVERY - Lucene on search1002 is OK: TCP OK - 0.026 second response time on port 8123 [02:56:09] PROBLEM - Lucene on search1001 is CRITICAL: Connection refused [02:56:13] see y'all tomorrow [03:03:31] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2634 [03:04:04] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2475 [03:04:04] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2475 [03:04:21] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2476 [03:04:21] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2476 [03:04:35] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2477 [03:04:35] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2477 [03:04:48] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2478 [03:04:49] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2478 [03:05:04] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2479 [03:05:04] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2479 [03:05:21] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2623 [03:05:36] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2635 [03:05:54] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2480 [03:05:55] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2634 [03:05:55] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2623 [03:05:55] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2635 [03:05:56] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2480 [04:21:57] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:23:18] New review: Tim Starling; "Your C skills are improving rapidly, so in this review I included some comments about style and conv..." [analytics/udp-filters] (refactoring); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2626 [04:23:18] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [05:13:55] !log killing purgeParserCache.php since it is probably doing more harm than good [05:13:58] Logged the message, Master [05:14:30] TimStarling: stop harming the servers! [05:21:48] !log on db40: giving the innodb manual the benefit of the doubt and following its advice, setting innodb_max_purge_lag to 100M, which should give a delay of 4.5ms [05:21:50] Logged the message, Master [05:22:12] I expect this will just instantly bring the site down, but who knows? [05:25:58] come on, when has the manual ever let you down? [05:25:59] !log on db40: purge lag is still increasing at 108 per second, so reducing innodb_max_purge_lag to 50M [05:26:01] Logged the message, Master [05:28:38] http://dev.mysql.com/doc/refman/5.1/en/innodb-multi-versioning.html [05:28:46] that's the relevant manual section [05:29:00] I can't see how it would make a difference if purging is single threaded, but what do I know? [05:29:13] anyway, 50M, still not making a difference [05:30:24] site's not down, purging's not happening more quickly [05:32:06] it's 10M now, that should be a 45ms delay [05:32:48] no, 90ms [05:35:21] !log on db40: reduced to 10M, should be causing massive delays, but the site's not down and the purge rate is lower if anything. Going to disable the mysql parser cache entirely. [05:35:23] Logged the message, Master [05:36:14] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=MySQL%20pmtpa&h=db40.pmtpa.wmnet&v=3&m=mysql_created_tmp_tables&r=hour&z=default&jr=&js=&st=1329456868&vl=tables&ti=mysql_created_tmp_tables&z=large [05:36:35] I don't believe anything those graphs say [05:36:38] lol [05:36:40] they seem to jump around randomly [05:37:04] why does it say "tables" not "tables per second" or something? [05:37:22] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=MySQL%20pmtpa&h=db40.pmtpa.wmnet&v=2&m=mysql_aborted_connects&r=hour&z=default&jr=&js=&st=1329457016&vl=conns&ti=mysql_aborted_connects&z=large [05:37:27] it must divide the counter by something, and I reckon there's a bug in it and it divides it by some randomly changing number [05:37:27] do you believe that? ;) [05:37:42] nope [05:38:04] I spent a long time staring at those graphs yesterday [05:38:06] !log tstarling synchronized wmf-config/CommonSettings.php [05:38:08] Logged the message, Master [05:38:19] before I eventually decided that they were rubbish and that I should look at other kinds of data [05:38:58] SHOW TABLE STATUS? [05:39:12] !log tstarling synchronized wmf-config/InitialiseSettings.php [05:39:14] Logged the message, Master [05:39:31] SHOW TABLE STATUS was useful, but SHOW INNODB STATUS is where the issue actually was [05:42:09] ok, now the lag is going in the right direction [05:42:39] so are all the db graphs bad? [05:42:47] that sounds like something that should be fixed [05:43:05] everything starting with mysql_ comes from the same script [05:43:45] purging 146 transactions per second now [05:44:01] at that rate it will only take 7.5 days to get down to zero [05:46:33] how many words will Evelyn learn by then? [05:49:13] SET GLOBAL eedeebee_ba_ba_gaaag = 100000000; [05:51:03] ok, so ehcache = suck, mysql = suck [05:51:05] what's left? [05:51:15] hadoop? [05:53:00] For thinks like DPL cache, what would be best? [05:53:31] I think there are 3 extensions that will still need NFS unless converted over [05:54:30] will they be hard to convert? I think I surveyed extensions a long time ago and it didn't look too hard [05:57:07] * Aaron|home looks again [05:57:20] like, before FileBackend [06:00:56] * Aaron|home sees DPL, Math, ConfirmEdit [06:03:04] aside from math, most of these just need a lightweight file cache [06:03:22] have three redundant copies with swift seems like overkill [06:03:43] if we get a better disk-based parser cache backend, it might be nice to use it for these [06:04:36] what was wrong with ecache? I keep forgetting [06:14:17] single-threaded disk I/O [06:16:08] * Aaron|home reads http://ehcache.org/ [06:25:44] Aaron|home yo [06:26:16] I had a rename request on your talk page :/ [06:29:21] I only see one request by someone and it was done [06:29:51] hmm, well there is the usurpation one [06:30:51] but that's another user [06:59:00] it seems like I can't read a NoSQL article without Domas being mentioned [06:59:13] heh [07:01:21] http://highscalability.com/blog/2011/7/25/is-nosql-a-premature-optimization-thats-worse-than-death-or.html [07:02:18] Aaron|home: hmm i'm surprised the whitecat move didn't happen when aerial did it server side and had to be done manually on wiki =/ thought that would have been cluster wide unless it missed it somehow [07:30:05] Aaron|home I am that other user :p [07:30:15] the japanese characters :) [07:38:21] PROBLEM - LVS Lucene on search-pool1.svc.pmtpa.wmnet is CRITICAL: Connection timed out [07:40:45] RECOVERY - LVS Lucene on search-pool1.svc.pmtpa.wmnet is OK: TCP OK - 0.001 second response time on port 8123 [07:41:21] PROBLEM - Lucene on search9 is CRITICAL: Connection timed out [07:53:03] PROBLEM - Lucene on search3 is CRITICAL: Connection timed out [07:55:18] RECOVERY - Lucene on search3 is OK: TCP OK - 0.001 second response time on port 8123 [07:55:36] RECOVERY - Lucene on search9 is OK: TCP OK - 0.004 second response time on port 8123 [09:05:39] PROBLEM - Disk space on mw43 is CRITICAL: DISK CRITICAL - free space: /tmp 32 MB (1% inode=87%): [09:06:06] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:07:18] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [09:09:33] RECOVERY - Disk space on mw43 is OK: DISK OK [09:54:42] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [09:57:42] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [10:00:42] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [10:00:42] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [10:45:06] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [10:46:00] RECOVERY - Lucene on search15 is OK: TCP OK - 0.002 second response time on port 8123 [10:49:45] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [10:53:30] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.010 second response time on port 8123 [10:54:15] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [10:56:39] RECOVERY - Lucene on search15 is OK: TCP OK - 2.995 second response time on port 8123 [11:10:00] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:21] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [11:35:30] PROBLEM - Disk space on mw48 is CRITICAL: DISK CRITICAL - free space: /tmp 70 MB (3% inode=87%): [11:40:35] Hullo: Is there a problem with watchlists currently? [11:41:48] * abartov has been unable to add pages to his watchlist on Meta, for more than an hour now [11:46:00] RECOVERY - Disk space on mw48 is OK: DISK OK [11:47:32] anyone? [11:51:43] abartov, here too [11:51:56] perhaps it was being discussed yesterday somewhere, lemmy search [11:52:17] nemo_bis: thanks! Guillom just confirmed too. [11:53:13] hm, no, it was something different [11:53:33] I'm reporting it on [[m:Talk:Wikimedia maintenance notice]] and I'll see if it's in bugzilla already [11:54:58] couldn't find it with a quick search [11:55:04] ok [11:55:29] FYI reported at https://meta.wikimedia.org/wiki/Talk:Wikimedia_maintenance_notice#Watchlist_issue [11:58:07] ah https://meta.wikimedia.org/w/index.php?title=Language_committee/Status/wp/nso&action=watch&token=d58a9e8f8e1ec2e19500a0bda5c5a8d0%2B%5C [11:58:22] wasn't the token thing added recently [11:58:55] it works without the trailing slash [11:58:59] filing a bug [11:59:28] thanks [11:59:47] Nemo_bis, feel free to reuse the text I posted on meta [11:59:53] ok [12:02:55] guillom, abartov, apergos https://bugzilla.wikimedia.org/show_bug.cgi?id=34469 [12:03:27] ok [12:03:58] great [12:10:07] zzz =_= [12:20:26] An error occurred while changing your watchlist settings for "Wikimedia Conference 2012/Program brainstorming". ... on metawiki [12:20:55] tried it on https: and http: [12:21:27] what's the url for read only access to git ? [12:22:00] e.g. for ssh://oren@gerrit.wikimedia.org/mediawiki/tools/mwdumper.git [12:23:28] DerHexer: We are aware of the watchlisting issues [12:24:01] DerHexer, https://bugzilla.wikimedia.org/show_bug.cgi?id=34469 [12:24:03] okay, thanks for the information! [12:24:14] hoping it wasn't a duplicate... [12:35:39] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [12:39:42] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [12:41:39] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [13:35:03] PROBLEM - Disk space on srv233 is CRITICAL: DISK CRITICAL - free space: /tmp 46 MB (2% inode=89%): [13:39:06] RECOVERY - Disk space on srv233 is OK: DISK OK [13:46:36] PROBLEM - Disk space on mw40 is CRITICAL: DISK CRITICAL - free space: /tmp 65 MB (3% inode=87%): [13:49:09] RECOVERY - Disk space on mw40 is OK: DISK OK [14:03:11] is there something wrong with irc.wikimedia.org today? [14:08:01] I haven't heard of anything [14:14:24] ok. thanks apergos. for some reason, I am not able to connect to the server [14:14:32] hmmm [14:15:18] I've just logged in fine [14:15:19] I'm already connected but [14:15:34] it does seem to let me join channels just fine [14:18:07] Wikimedia heavily relies on load balancing, using LVS with PyBal [14:18:13] What is PyBal ? [14:18:39] http://svn.wikimedia.org/viewvc/mediawiki/trunk/pybal/ [14:18:54] written in house by mark [14:19:36] PROBLEM - Disk space on srv270 is CRITICAL: DISK CRITICAL - free space: /tmp 42 MB (2% inode=88%): [14:19:39] thanks [14:19:51] sure [14:37:45] !log hashar synchronized php-1.19/LocalSettings.php 'syslog debug statement to investigate TESTWIKI issue' [14:37:47] Logged the message, Master [14:39:33] RECOVERY - Disk space on srv270 is OK: DISK OK [14:40:49] !log hashar synchronized php-1.19/LocalSettings.php 'debug statement with argv join' [14:40:51] Logged the message, Master [14:58:36] !log hashar synchronized php-1.19/LocalSettings.php [14:58:38] Logged the message, Master [14:59:16] !log reedy synchronized php-1.19/includes/specials/SpecialDeletedContributions.php 'r111752' [14:59:18] Logged the message, Master [15:02:13] !log hashar synchronized php-1.19/LocalSettings.php 'make jobrunners for testwiki to use the apache CommonSettings file instead of the non existant /home/wikipedia one' [15:02:14] Logged the message, Master [16:06:29] New patchset: Pyoungmeister; "adding raid5 setup for searchidx boxxies" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2637 [16:06:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2637 [16:29:39] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [16:34:04] New review: RobH; "looks right, but partman is tricky" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2637 [16:34:05] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2637 [16:39:51] PROBLEM - Disk space on mw8 is CRITICAL: DISK CRITICAL - free space: /tmp 9 MB (0% inode=86%): [16:44:03] PROBLEM - Disk space on mw16 is CRITICAL: DISK CRITICAL - free space: /tmp 67 MB (3% inode=86%): [16:49:18] RECOVERY - Disk space on mw16 is OK: DISK OK [16:49:29] New patchset: Pyoungmeister; "increasing timeout on check_lucene, at least temporarily, to attempt to combat flapping" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2638 [16:49:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2638 [16:50:20] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2638 [16:50:20] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2638 [17:03:42] RECOVERY - Disk space on mw8 is OK: DISK OK [17:48:51] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:49:05] http://pastebin.com/GGz9hkSP [17:49:17] I get this error from wikipedia when I try to merge accounts [17:51:15] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [17:51:25] should I panic? [17:52:44] ToAruShiroiNeko, wich wikipedia ? [18:19:25] !log aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php 'force index' [18:19:26] Logged the message, Master [18:22:03] !log aaron synchronized php-1.18/includes/api/ApiQueryAllUsers.php [18:22:05] Logged the message, Master [18:40:27] PROBLEM - Disk space on mw9 is CRITICAL: DISK CRITICAL - free space: /tmp 6 MB (0% inode=86%): [18:41:48] RECOVERY - Disk space on mw9 is OK: DISK OK [18:43:16] !log reedy synchronized php-1.18/extensions/WikimediaMessages/ [18:43:18] Logged the message, Master [18:44:13] !log reedy synchronized php-1.19/extensions/WikimediaMessages/ [18:44:15] Logged the message, Master [18:49:54] PROBLEM - ps1-d2-pmtpa-infeed-load-tower-A-phase-Y on ps1-d2-pmtpa is CRITICAL: ps1-d2-pmtpa-infeed-load-tower-A-phase-Y CRITICAL - *2525* [18:50:10] !log reedy synchronized wmf-config/ 'Sync ExtensionMessages' [18:50:12]