[00:34:34] I know I'm indef'd, but Due to high database server lag, changes newer than 217 seconds may not appear in this list. [00:39:01] 5 minute replag - known issue? [00:39:19] "Due to high database server lag, changes newer than 417 seconds may not appear in this list. " [00:39:27] Yep, that [00:43:07] Now 10 minutes [00:47:32] That's enwp, btw [00:48:48] evening all. Anyone aware of any large jobs or work being done on databases please? Seems to be a db lag going up in quite large numbers, was 693 seconds a few moments ago, now up to almost 900 seconds on contributions, but nowhere else it seems. [00:50:11] yep, it's going up at a rate of knots, now 1028 seconds. [00:52:31] We're at 20 minutes... [00:53:00] its just db12 [00:53:12] Is it not replicating? [00:53:34] Coalballs is mentioning that they're getting nginx errors on their watchlist too - 504 Gateway Timeout [00:53:37] that is the only server lagging [00:54:53] "Due to high database server lag, changes newer than 1,259 seconds may not appear in this list." [00:54:53] =D [00:54:55] 10 appears to have started lagging too. [00:55:11] it's only a minute down at the moment though [00:56:38] well that's caught up, so it's just db12 behind, as Betacommand mentioned. [00:57:15] It's catching up now. [00:57:40] @replag [00:57:43] Nemo_bis: [s1] db12: 844s [00:57:50] ugh [00:58:06] Nemo_bis: That was 1300 a minute ago. [00:59:18] 861 seconds according to nagios.wikimedia.org, but that is going down significantly, was 1102s the last time it refreshed :) [00:59:42] Someone must have been flushing too much toilet paper down the pipes and blocked it somewhere :P [01:00:11] http://ganglia.wikimedia.org/latest/?c=MySQL%20pmtpa&h=db12.pmtpa.wmnet&m=load_one&r=hour&s=by%20name&hc=4&mc=2 [01:00:45] oh, high replay within the cluster? that's rare [01:00:47] @replag all [01:00:49] Krinkle: [s1] db38: 0s, db36: 0s, db32: 0s, db59: 0s, db60: 0s, db12: 0s; [s2] db52: 0s, db53: 0s, db54: 0s, db57: 0s; [s3] db39: 0s, db34: 0s, db25: 0s, db11: 0s [01:00:50] ?? http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=MySQL%20pmtpa&h=db12.pmtpa.wmnet&v=14574&m=mysql_innodb_history_list&r=hour&z=default&jr=&js=&st=1336870780&vl=length&ti=mysql_innodb_history_list&z=large [01:00:50] Krinkle: [s4] db31: 0s, db22: 0s, db33: 0s, db51: 0s; [s5] db35: 0s, db45: 0s, db44: 1s, db55: 0s; [s6] db43: 0s, db47: 0s, db50: 0s; [s7] db37: 0s, db56: 0s, db58: 0s, db26: 0s [01:01:12] And, we're back. [01:01:44] rep delay down to 52 seconds, should be nothing left of it by now :) [01:02:10]

RECOVERY - MySQL Replication Heartbeat on db12 is OK: OK replication delay 0 seconds [01:02:12] @replag [01:02:13] yeah [01:02:14] Krinkle: No replag currently. See also "replag all". [01:03:25] Does nagios-wm not visit this channel anymore? [01:04:31] last seen on network 2 years 8 weeks ago :) [01:04:43] Mar 27 20:27:40 * nagios-wm has quit (Remote host closed the connection) [01:04:44] Krenair, no [01:05:05] only #wikimedia-operations, people were sick of double reading [01:05:06] http://ganglia.wikimedia.org/latest/?c=PDF%20servers%20pmtpa&h=pdf1.wikimedia.org&m=network_report&r=hour&s=by%20name&hc=4&mc=2 [01:05:19] Nemo_bis, is that a closed channel? [01:05:26] no [01:05:37] but not one to be flooded [01:05:42] by humans I mean :p [01:06:02] so bots only, no talking, is that about right? [01:06:12] noo [01:06:35] Looks like roughly 50 people, 4 bots [01:06:55] Unmoderated [01:06:58] http://ganglia.wikimedia.org/latest/?c=API%20application%20servers%20pmtpa&h=srv256.pmtpa.wmnet&m=network_report&r=hour&s=by%20name&hc=4&mc=2 [01:07:31] Magog_the_Ogre: ping [01:07:45] I will sit, I will watch, I will say absolutely nothing :) [01:08:10] I noticed the status here though, saying that you were "back to normal" - had you been doing maintenance or something? [01:11:16] * jeremyb hands TheWoozle a vimtutor [01:11:24] gah [01:11:33] THO|Cloud* [01:11:51] May 03 22:49:09 * saper has changed the topic to: Technical help for Wikimedia wikis: https://meta.wikimedia.org/wiki/Tech | Status: back to normal | MediaWiki: #mediawiki | Toolserver: #wikimedia-toolserver | Pastebin: http://p.defau.lt/ | Server admin log: http://bit.ly/wikisal | Channel is logged: http://goo.gl/mRajX [01:14:12] Cat_in_the_Hat, better? [01:15:20] Yes, thank you :) [01:15:59] I just wondered what you'd been doing, it's nice to know sometimes, what kind of grief goes on at the backend. We only see the front :) [01:17:52] Me? I'm just a random user [01:18:08] I'm as uninvolved in that sort of stuff as you are [02:24:51] !log LocalisationUpdate completed (1.20wmf2) at Sun May 13 02:24:51 UTC 2012 [02:24:57] Logged the message, Master [11:55:20] just a yes-no question: is the enwiki dump download server denying android user agents and giving them 403 forbidden errors? [12:05:13] blackbit: Works for me, so no [12:05:25] hoo|busy, thanks for trying [12:06:03] (I just want to add that the error shows me when I enter into the folder structure, not just by 'visiting' the url) [12:17:41] Hi I am looking at http://chr.wikipedia.org/wiki/Special:ActiveUsers [12:17:49] and a lot of bot edits show [12:18:06] I am guessing code does not count global bots as bots unless they have a local flag [12:18:11] is this correct? [12:56:19] it seems so [12:56:32] "Skip bots" is skipping users in the bots group [12:56:41] sorry, 'bot' group [12:56:53] group for global bots is 'Global_bot' [12:59:44] it should skip both IMHO [12:59:54] since most bots are global these days [13:01:34] it may be a bit ugly, but easy enough to add a skip Global_bot line there [13:01:40] can you add a bug? [13:02:11] sure [13:05:55] done [13:06:03] https://bugzilla.wikimedia.org/show_bug.cgi?id=36812 [13:07:19] I made a few more suggestions beyond global bots [14:00:28] Platonides could you take a look at the bug again I proposed further enhancements :/ [16:47:15] * ToAruShiroiNeko pokes Platonides [17:08:49] heyho, does anyone know what happened to the "2011-christmas-stats"? [17:08:50] http://dumps.wikimedia.org/other/pagecounts-raw/2011/2011-12/ [17:09:30] the files for access logs from Dec. 23rd till 26th are empty ;( [17:10:36] does anyone happen to have a backup including them? [18:04:21] *** This bug has been confirmed by popular vote. *** [18:04:22] :-O [18:05:15] :O [18:10:47] I suspect this is not working: https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&list_id=115455&field0-0-0=everconfirmed&type0-0-0=equals&value0-0-0=1&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=DUPLICATE&resolution=WORKSFORME&product=Analytics&product=CiviCRM&product=Cortado&product=Datasets&product=Huggle&product=Kate%27s%20Tools&product=Logwood&product=MediaWiki&product=Med [18:10:48] iaWiki%20extensions&product=mwEmbed&product=Tools&product=Wikimedia&product=Wikimedia%20Labs&product=Wikimedia%20Mobile&product=Wikipedia%20App&product=Wiktionary%20App&product=Wiktionary%20tools [18:10:51] whoops [18:10:57] anyway, gives all bugs [20:12:16] bonne nuit~ [21:55:23] gn8 folks [21:55:54] DaBPunkt: 'night