[06:20:12] <GerardM->	 the replication of Wikidata to labs is still broken ... day 3
[06:27:38] <TimStarling>	 GerardM-: is anyone working on it?
[06:28:27] <GerardM->	 no
[06:28:51] <GerardM->	 people are saying things like "Thanksgiving"
[06:29:38] <GerardM->	 it however affects the tooling we are building to make Wikidata more attractive
[06:30:36] <GerardM->	 yesterday evening they were trying to restart a webservice without much success as far as I could tell
[06:31:01] <GerardM->	  lighttpd was a magic word they used
[06:33:26] <GerardM->	 I will be grateful when it works again
[06:50:20] <marktraceur>	 GerardM-: What's broken? Any clues?
[06:51:08] <GerardM->	 there is a process that regularly updates the labs Wikidata from production Wikidata
[06:51:22] <GerardM->	 it has not been working for the third day now
[06:51:45] <marktraceur>	 I get that, is there any error message or code you think has broken it?
[06:52:02] <marktraceur>	 I don't even know what repo to look at, maybe we could start there
[06:52:37] <GerardM->	 this is what Magnus said about it: "s5 replag is bad: http://ganglia.wmflabs.org/latest/?c=tools&h=tools-login&m=load_one&r=hour&s=by%20name&hc=4&mc=2
[06:52:38] <GerardM->	 last update in the replicated database at this moment: 2013-11-27 00:33:03"
[06:52:58] <GerardM->	 does that help ?
[06:53:42] <GerardM->	 and ... "more stats on s5: http://ganglia.wmflabs.org/latest/graph_all_periods.php?c=tools&h=tools-login&v=142921&m=s5&r=hour&z=default&jr=&js=&st=1385655305&z=large"
[07:02:31] <marktraceur>	 Hrm
[07:02:46] <marktraceur>	 Graphs are pretty but don't really help me pinpoint an issue
[07:02:54] <marktraceur>	 Especially without code for the process
[07:05:22] <GerardM->	 last time I heard, labs was a grey area when it comes to responsibility
[07:05:41] <marktraceur>	 Heh
[07:05:45] <marktraceur>	 Toollabs especially
[07:05:49] <GerardM->	 I also heard that it was an improvement over the toolserver
[07:05:51] <marktraceur>	 Hah, great time for Ryan to log out
[07:06:07] <GerardM->	 trying to be fair
[07:58:06] <GerardM->	 marktraceur I thought that the labs configuration was also in puppet ..
[07:58:37] <GerardM->	 is the purpose of that not that it is easy to reestablish a working order ?
[07:59:03] <marktraceur>	 Well
[07:59:15] <marktraceur>	 With some idea of what code is being problematic, I could look at recent changes.
[08:00:28] <GerardM->	 there was this German problem a few days ago ... it broke a lot of things
[08:01:01] <GerardM->	 and it was acknowledged that the person "fixing" things did not really know the extend of the damage
[08:01:29] <GerardM->	 he said in the labs mail ... please restart what did not restart
[08:08:28] <GerardM->	 http://lists.wikimedia.org/pipermail/labs-l/2013-November/001883.html
[08:09:27] <marktraceur>	 Aha, maybe Coren knows then.
[08:09:35] * marktraceur  will let him sort that out and go to bed

[09:00:09] <GerardM->	 thanks marktraceur 
[10:19:50] <Romaine>	 is there a disturbance on the wiki, some wikisyntax isn't working
[10:19:55] <Romaine>	 or an update?
[10:21:08] <p858snake|l>	 descriptive is descriptive
[10:22:53] <Reedy>	 Romaine: wtf?
[10:23:29] <Romaine>	 somehow the cliackable buttons do not work on nl-wiki, while they do work on Commons
[10:26:31] <Reedy>	 what clickable buttons?
[10:31:19] <Romaine>	 like the green one on the bottom: https://commons.wikimedia.org/wiki/Commons:First_steps#footer
[10:32:32] <Reedy>	 any JS errors on nlwiki?
[10:33:17] <Romaine>	 unknown
[10:33:37] <Reedy>	 having a look would be useful
[10:35:29] <Romaine>	 I don't know how to look up?
[10:36:00] <Romaine>	 where should I see that?
[10:37:58] <Romaine>	 here should we get that same coloured button: https://nl.wikipedia.org/wiki/Wikipedia:Start
[10:38:00] <Reedy>	 https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CC8QFjAA&url=http%3A%2F%2Fwebmasters.stackexchange.com%2Fquestions%2F8525%2F&ei=hG6YUqAf5-bsBtyggfgC&usg=AFQjCNHdxTWIPgrwgyBJLCZGp8-z7_G4aQ&sig2=TlepX2ffX2nnlddNmNY9wg&bvm=bv.57155469,d.ZGU
[10:38:13] <Reedy>	 fail
[10:38:22] <Reedy>	 http://webmasters.stackexchange.com/questions/8525/how-to-open-the-javascript-console-in-different-browsers
[10:38:27] <Reedy>	 I don't see a button on commons
[10:38:48] <Reedy>	 uh, not at the bottom anyway
[10:39:20] <Romaine>	 mmm, then it is a multiple wiki issue
[10:39:35] <Romaine>	 perhaps because the software was updated in the past day?
[10:39:47] <Reedy>	 It wasn't
[10:39:58] <Reedy>	 I see the "begin the tutorial" but it's near the top on commons
[10:40:27] <Romaine>	 mmm
[10:40:41] <Romaine>	 but you get a green bar with text in white in it?
[10:40:54] <Romaine>	 saying: begin the tutorial
[10:41:04] <Reedy>	 yeah
[10:41:57] <Romaine>	 that color is gone
[10:42:03] <Romaine>	 on nl-wiki
[10:43:02] <Romaine>	 like on https://nl.wikipedia.org/wiki/Wikipedia:Start
[10:43:14] <Romaine>	 there a green and red bar should appear
[10:43:20] <Reedy>	 site css change?
[10:43:26] <Romaine>	 if I checkl a diff the colors are shown
[10:43:33] * Romaine  goes looking

[10:44:23] <Romaine>	 not something what should effect this I would say
[10:45:26] <Romaine>	 https://nl.wikipedia.org/w/index.php?namespace=8&tagfilter=&title=Speciaal%3ARecenteWijzigingen
[10:46:39] <Romaine>	 have you any idea what causes this?
[11:12:38] <ori-l>	 the module isn't loaded
[11:13:04] <ori-l>	 the styles are provided by jquery ui
[11:13:25] <ori-l>	 mw.loader.getState('jquery.ui.button') === 'ready' on enwiki, 'registered' (i.e., not loaded) on nlwiki
[11:15:13] <ori-l>	 see http://www.mail-archive.com/wikitech-ambassadors@lists.wikimedia.org/msg00449.html
[11:15:17] <ori-l>	 ^ Romaine
[11:15:53] <ori-l>	 'If your wiki uses jquery.ui buttons in wikitext, the styles will not be loaded unless something explicitly loads or depends on jquery.ui.'
[11:22:48] <ori-l>	 Romaine: add this to Common.js: if ( mw.config.get('wgPageName') === "Wikipedia:Start/Welkom" ) mw.loader.using('jquery.ui.button');
[11:23:36] <Romaine>	 ok
[11:23:55] <Romaine>	 but it are several pages where we use it
[11:25:19] <ori-l>	 commonswiki has a more robust solution that loads the module if the elements that require it are detected
[11:25:21] <ori-l>	 https://commons.wikimedia.org/wiki/MediaWiki:Common.js
[11:25:41] <ori-l>	 this is the 'jQuery UI loader' bit from rillke
[11:26:56] <Romaine>	 I should copy the whole parte from jQuery UI loader to the bottom of that page?
[11:28:22] <ori-l>	 Romaine: just https://dpaste.de/34VM/raw
[11:32:04] <Romaine>	 ori-l: doesn't work it seems
[11:32:17] * ori-l  checks

[11:33:24] <ori-l>	 Romaine: i don't see it on http://nl.wikimedia.org/wiki/MediaWiki:Common.js ?
[11:33:43] <ori-l>	 oh, sorry, you misunderstood me
[11:33:52] <ori-l>	 it doesn't go on the target page; it goes in MediaWiki:Common.js
[11:33:53] <Romaine>	 yes
[11:34:04] <ori-l>	 or rather, I misunderstood you
[11:34:14] <ori-l>	 sorry for the confusion
[11:34:15] <Romaine>	 it is added here: https://nl.wikipedia.org/wiki/MediaWiki:Common.js
[11:35:56] <Romaine>	 bottom
[11:36:31] <ori-l>	 it works now
[11:36:40] <ori-l>	 just took a couple of minutes for the resourceloader cache to clear
[11:36:47] <Romaine>	 ok
[11:37:04] <Romaine>	 works
[11:37:07] <ori-l>	 :)
[11:37:32] <Romaine>	 thank you very much!
[11:37:55] <ori-l>	 no problem
[11:40:28] <Romaine>	 is there any reason why it worked before and suddenly not any more (until this fix)?
[11:45:35] <Reedy>	 Romaine: Yes
[11:45:45] <Reedy>	 [11:15:10] <ori-l> see http://www.mail-archive.com/wikitech-ambassadors@lists.wikimedia.org/msg00449.html
[11:46:49] <ori-l>	 jQuery UI is a large library; AFAIK it was never anyone's explicit intention for it to load on every single page
[11:47:03] <ori-l>	 but at some point some extension started adding it by default on every page, and people got used to it being there
[11:52:19] <Romaine>	 ah ok
[11:52:21] <Romaine>	 good to know
[13:59:55] <Coren>	 GerardM-: FYI, dewiki is back.
[14:00:40] <GerardM->	 coren, does that mean that the replication of Wikidata is working as well ?
[14:00:48] <Coren>	 GerardM-: Sean fixed the remaining issue last night and it caught up since.
[14:01:07] <Coren>	 GerardM-: It should; they are on the same shard and the replication status is completely synced.
[14:01:34] <GerardM->	 Ok let me find out
[14:14:56] <GerardM->	 it is :)
[14:15:03] <GerardM->	 Happy :)
[14:18:14] <GerardM->	 I have changed an item ... it should be replicated in half an hour ... that will make it 100% obvious .. Q3523640
[14:34:29] <GerardM->	 Hmmm I have waited almost twenty minutes there is no replication
[14:35:16] <GerardM->	 Coren: how long should it take maximally ?
[14:36:18] <GerardM->	 15:03, 29 November 2013 is when I added the label for Dutch
[14:41:48] <Coren>	 GerardM-: Ah, you're right -- it caught up to the 27th but no further.
[14:41:51] * Coren  ponders.

[14:42:10] <Coren>	 But the slave is up to date with its master, so perhaps it's one level up.  I shall go check.
[19:42:39] <GerardM->	 does anyone know if the replication has started again ?
[19:42:50] <GerardM->	 from Wikidata to labs
[19:47:27] <YuviPanda>	 GerardM-: the two people who would know are Coren and springle, both of whose nicks say 'afk' atm
[19:48:00] <GerardM->	 coren did look at it ... and the last I heard was that he had lo check one level higher
[19:48:16] <GerardM->	 that was five hours ago
[19:48:35] <YuviPanda>	 GerardM-: indeed, and one level higher is usually springle
[19:49:01] <GerardM->	 AH
[19:49:17] <YuviPanda>	 GerardM-: banging on labs-l might be your best option for now :)
[19:49:51] <GerardM->	 <grin> I got better results here
[19:51:23] <YuviPanda>	 GerardM-: heh