[00:58:49] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Remove unused wmgUseUsabilityInitiativeAlpha'
[00:58:51] <morebots>	 Logged the message, Master
[00:59:31] <logmsgbot>	 !log reedy synchronized wmf-config/CommonSettings.php  'Remove wmgUsabilityPrefSwitch'
[00:59:33] <morebots>	 Logged the message, Master
[01:00:16] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Remove wmgUsabilityPrefSwitch'
[01:00:17] <morebots>	 Logged the message, Master
[01:02:39] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Remove some dupe code'
[01:02:41] <morebots>	 Logged the message, Master
[01:13:50] <logmsgbot>	 !log reedy synchronized wmf-config/CommonSettings.php  'Remove more crap'
[01:13:52] <morebots>	 Logged the message, Master
[02:17:57] <logmsgbot>	 !log LocalisationUpdate completed (1.19) at Fri Mar 30 02:17:56 UTC 2012
[02:17:58] <morebots>	 Logged the message, Master
[06:39:21] <logmsgbot>	 !log tstarling synchronized php-1.19/includes/specials/SpecialBlock.php 
[06:39:22] <morebots>	 Logged the message, Master
[06:40:35] <logmsgbot>	 !log tstarling synchronized php-1.19/includes/specials/SpecialBlock.php 
[06:40:37] <morebots>	 Logged the message, Master
[06:41:33] <logmsgbot>	 !log tstarling synchronized php-1.19/includes/specials/SpecialBlock.php 
[06:41:35] <morebots>	 Logged the message, Master
[07:03:45] <TimStarling>	 !log running bug 35578 cleanup script in screen on fenari
[07:03:47] <morebots>	 Logged the message, Master
[10:07:49] <nosy>	 hello
[10:08:20] <nosy>	 i am having replication problems with db47
[10:08:35] <nosy>	 yesterday db47 hung but after reboot everything was fine
[10:08:47] <nosy>	 now replication stopped and i cannot find a reason
[10:09:17] <nosy>	 i can reach the master, the tunnel is online, my slave mysql process does not complain but still the replag increases
[10:15:12] <jeremyb>	 nosy: can you pastebin `show slave status\G` ?
[10:16:15] <nosy>	 jeremyb: here you go: http://pastebin.com/PyQ7aQZY
[10:17:28] <jeremyb>	 also, do you have a timestamp for reboot? a graph of your replag?
[10:17:52] <nosy>	 for the reboot of db47? hang on
[10:18:01] * jeremyb  prefers http://pastebin.com/raw.php?i=PyQ7aQZY ;)

[10:18:04] <jeremyb>	 yes please
[10:18:23] <nosy>	 our replication: http://munin.toolserver.org/Database/z-dat-s6-a/mysql_replication.html
[10:18:58] <nosy>	 and http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&m=&c=MySQL+pmtpa&h=db47.pmtpa.wmnet&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS
[10:34:04] <nosy>	 jeremyb: can you find any hint in the mysql log of db47?
[10:45:45] <jeremyb>	 nosy: i don't have access to that. and the wifi sucks here (that's why I disappeared) and now I've been tasked to do notetaking for this meeting
[10:46:03] * jeremyb  hopes someone else shows up... apergos ?

[10:46:20] <jeremyb>	 or mutante
[10:46:24] <jeremyb>	 bbl
[10:46:26] <nosy>	 jeremyb: ok seems i have to bug people here later :D
[10:46:54] <jeremyb>	 gah
[10:48:51] * jeremyb  has summoned nosy ;)

[10:49:01] <jeremyb>	 30 10:48:18 <jeremyb> those 2 people I just mentioned (mutante and apergos) both have access and are recently active. one in the last 5 mins
[10:49:36] <jeremyb>	 (I *think*... a little fuzzy about ape rgos and i don't have time to ask puppet atm ;) )
[10:49:39] <jeremyb>	 bbl
[11:24:45] <jeremyb>	 good morning east coast
[11:25:17] <jeremyb>	 ^demon: do you know who can see server logs on db47?
[11:25:33] <^demon>	 Any root ;-)
[11:25:45] <^demon>	 I don't think mortals can.
[11:25:47] <jeremyb>	 so... not you then
[11:25:55] <nosy>	 mutante did alreadyhelp me but it was a strange problem
[11:26:00] <jeremyb>	 orly
[11:26:11] <jeremyb>	 (nvm ^demon)
[11:26:27] <nosy>	 our slave was much further with the replication master position then db47 was
[11:26:43] <nosy>	 probably this is related to the master position adjustment yesterday
[11:27:06] <jeremyb>	 I can definitely see it recovering
[11:27:22] <nosy>	 jeremyb: where?
[11:27:27] <jeremyb>	 munin
[11:27:36] <jeremyb>	 13k secs -> 1.3k secs
[11:27:37] <nosy>	 ours? yes
[11:27:38] <jeremyb>	 iirc
[11:27:57] <nosy>	 i rotated the log position backwards
[11:28:03] <nosy>	 120330 11:18:26 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='10.24.1.15', master_port='3006', master_log_file='db47-bin.000145', master_log_pos='608985486'. New state master_host='10.24.1.15', master_port='3006', master_log_file='db47-bin.000145', master_log_pos='601409784'.
[11:28:21] <nosy>	 mutante found the position the master process of db47 had
[11:28:32] <nosy>	 but i think it can be fishy
[11:28:40] <nosy>	 we might have inconsistent data
[11:28:53] <jeremyb>	 is this the one that was already getting a fresh dump?
[11:28:57] <jeremyb>	 @info db47
[11:28:58] <dbbot-wm>	 jeremyb: [db47: s6] 10.0.6.57
[11:28:59] <nosy>	 nope
[11:29:09] <nosy>	 do i get a flat rate on this or what :D
[11:29:11] * jeremyb  pats dbbot-wm

[11:29:20] <jeremyb>	 heh
[11:29:22] <mutante>	 i could not really see problems on db47, replication lag ok
[11:29:38] <mutante>	 and in full processlist i saw this state 
[11:29:43] <mutante>	 State: Has sent all binlog to slave; waiting for binlog to be updated
[11:29:56] <jeremyb>	 right, but there's multiple slaves?
[11:29:59] <mutante>	 associated with the slave amaranth.toolserver
[11:30:08] <jeremyb>	 huh
[11:30:15] <jeremyb>	 also, that just means it was sent
[11:30:40] <mutante>	 yeah, like we sent everything but waiting for amaranth
[11:30:41] <jeremyb>	 mysql replication works by sending the log and it's written on the slave in one thread
[11:31:12] <jeremyb>	 a different thread does reading from the copy on local disk and executing off it
[11:31:24] <jeremyb>	 both threads were running according to nosy's paste
[11:32:03] <jeremyb>	 it does sound fishy to me. maybe there were some missed entries between the 2 log entries
[11:32:16] <jeremyb>	 and so maybe inconsistency
[11:32:46] <jeremyb>	 I would have just done stop slave; start slave; first and seen what happened.
[11:33:23] <jeremyb>	 is lunch here now... bbl
[11:33:28] <nosy>	 jeremyb: i did that before i asked here
[11:33:32] <jeremyb>	 nosy: see you at the party tonight? (wmcon)
[11:33:35] <jeremyb>	 nosy: ahh
[11:33:47] <nosy>	 jeremyb: where?
[11:34:01] <nosy>	 berlin?
[11:34:07] <jeremyb>	 yes
[11:34:41] <nosy>	 oh where exactly?
[11:34:54] <jeremyb>	 between oranienburgerstr. and oranienburger tor on ziegelstr. baxpax.de "downtown"
[11:35:12] <nosy>	 ah wow i ll try thank you for the info
[11:35:56] <jeremyb>	 https://secure.wikimedia.org/wikipedia/meta/wiki/Wikimedia_Conference_2012/Schedule/Friday says 8pm
[11:36:13] <nosy>	 utc or what ;)
[11:36:28] <nosy>	 cet
[11:40:00] <mutante>	 oh.. "Berlin Underworld Tour" sounds mysterious but cool
[11:40:38] <mutante>	 i guess CEST
[16:20:36] <sumanah>	 Beau_: ialex Nemo_bis Please register for the Berlin Wikimedia & MediaWiki hackathon 1-3 June wmberlin.eventbrite.com Help available with visas & cost
[21:00:21] <Nemo_bis>	 Platonides, just read http://lists.wikimedia.org/pipermail/wikitech-l/2012-March/059530.html
[21:00:36] <Nemo_bis>	 do you agree with https://bugzilla.wikimedia.org/show_bug.cgi?id=35535#c2 then?
[21:53:43] <DaBPunkt>	 gn8 folks