[02:03:37] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by 210.22.145.214 link https://www.mediawiki.org/w/index.php?diff=878987 edit summary: [+5] 
[02:38:02] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=878988 edit summary: [-20] external link -> internal
[03:35:31] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by 1.38.16.80 link https://www.mediawiki.org/w/index.php?diff=878995 edit summary: [+20] 
[05:25:30] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=879023 edit summary: [-20] Undo revision 878995 by [[Special:Contributions/1.38.16.80|1.38.16.80]] ([[User talk:1.38.16.80|talk]])
[05:25:58] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=879024 edit summary: [-5] Undo revision 878987 by [[Special:Contributions/210.22.145.214|210.22.145.214]] ([[User talk:210.22.145.214|talk]])
[11:16:22] <GerardM>	 Coren are you around ?
[14:07:47] <Pratyya>	 Cyberpower678:  do you know how to send mass message?
[14:08:20] <Cyberpower678>	 Pratyya, no
[14:08:42] <Pratyya>	 anyone here knows? How to send mass message?
[14:09:09] <yuvipanda>	 Pratyya: ask legoktm
[14:09:30] <Pratyya>	 is he available atm? yuvipanda 
[14:09:41] <yuvipanda>	 mostly not
[14:10:06] <Pratyya>	 Then...
[14:20:26] <Coren>	 GerardM: Am now.
[14:22:01] <GerardM>	 Hoi ... I updated a bug twice today ... it would be really good if Magnus is able to continue the work he does
[14:23:11] <GerardM>	 Coren: I also suspect that the Wikidata synchronisation failed
[14:23:36] <Coren>	 GerardM: Linky?
[14:24:21] <GerardM>	 since I blogged http://ultimategerardm.blogspot.nl/2014/01/europeana-advertises-thomas-lawrence.html I added over 500 painters
[14:25:27] <GerardM>	 they are not coming to the query (that uses the synchronised data) http://208.80.153.172/wdq/?q=claim[106:1028181]
[14:26:05] <Coren>	 Could you link me to the bug?
[14:27:12] <GerardM>	 I just learned it is also bug nr 59007
[14:27:58] <GerardM>	 (chatting with Magnus)
[14:28:19] <GerardM>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=59007
[14:28:32] <Coren>	 Ah.  It's a non-tools project, that's why it didn't pop up in my general bug lists.  Hang on, that's probably gluster being broken.
[14:31:06] <Coren>	 GerardM: Should be fixed.  Sorry about the delay; between the holidays and travel for the new datacenter visits, that one slipped through the cracks.
[14:32:22] <GerardM>	 what does it take to make it a "tools" project ?
[14:32:23] <GerardM>	 Thanks
[14:32:59] <GerardM>	 but Coren how to make it a recognised tools project ? A link will do ... :)
[14:34:01] <Steinsplitter>	 omg
[14:34:20] <Coren>	 GerardM: It needs to be on 'tools' and not be a separate project.  :-)  All of that said, however, given how critical that has become perhaps it should be time to consider moving to production?
[14:35:24] <GerardM>	 <grin> is that not its own can of wurms ?
[14:36:19] <Coren>	 GerardM: Or we can discuss upping that project's status to "semi-production" like deployment-prep or tools; but that needs ops resources so it can only come from Erik (and, honestly, we are very short-staffed now)
[14:36:22] <JohannesK_WMDE>	 hi Coren. regarding this bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=49872 it has been re-broken since end of november, and we are getting mails from disgruntled users because the alg doesn't work for anything except enwiki. could you look into it or have someone look into it?
[14:39:18] <Coren>	 JohannesK_WMDE: I've assigned the bug to Sean (the DBA) with a note.  He should be able to fix this shortly.
[14:39:31] <GerardM>	 Coren does that mean that it has to be puppetised ?
[14:40:08] <Coren>	 GerardM: Yes it does; that is the most important factor for stable support since that allows change management and recovery.
[14:41:26] <scfc_de>	 Which tools are you talking about?  "Widar" seems to be located at http://tools.wmflabs.org/widar/.  Is there some backend to it?
[14:42:14] <GerardM>	 scfc_de: yes thgere is 
[14:42:22] <Coren>	 scfc_de: the wikidata-dev and whatever runs there.
[14:42:27] <GerardM>	 it uses the WDQ
[14:42:47] <GerardM>	 and the synchronisation from Wikidata has restarted
[14:42:55] <GerardM>	 has to catch up for over a week
[14:43:17] <Coren>	 GerardM: It's almost certainly the same cause; gluster broke which made the filesystem readonly.
[14:43:36] <Coren>	 GerardM: The good news is, there is no Gluster in eqiad.  :-)
[14:43:44] <GerardM>	 :)
[14:43:59] <Coren>	 GerardM: But seriously, if WDQ has become critical then we need to discuss real support.
[14:44:47] <Coren>	 Labs is great for dev work, but shouldn't be used for things on the critical path.
[14:45:07] <Coren>	 Well, not without puppetizing and documenting enough that ops can give support.
[14:46:06] <GerardM>	 I agree ... but the functionality is not WMF yet
[14:46:06] <GerardM>	 it is used as an extension to extended search in many Wikipedias
[14:46:12] <scfc_de>	 If WDQ is some query backend, it should probably go in an Extension/API.
[14:47:14] <GerardM>	 ... it is a fine line ... Magnus can learn puppet he said
[14:49:14] <scfc_de>	 Grepping for wdq and wikidata-wdq-mm in widar and magnustools comes up empty.
[14:49:29] <scfc_de>	 *wikidata-dev-mm
[14:52:48] <GerardM>	 scfc_de: that query backend is NOT WMF supported
[14:53:01] <GerardM>	 it is in anticipation of similar official functionality
[14:53:18] <Steinsplitter>	 OmgData :/
[14:54:58] <scfc_de>	 GerardM: I didn't say the opposite.
[14:56:04] <GerardM>	 scfc_de: how then is it to go into an extension API ?
[14:58:16] <scfc_de>	 GerardM: I said (in re extensions/APIs): "If WDQ is some query backend, it should probably go in an Extension/API."  I said (in re bug 59007): "Grepping for wdq and wikidata-wdq-mm in widar and magnustools comes up empty." aka: "I don't see why bug 59007 is holding up widar."
[15:01:22] <GerardM>	 it does not hold up Widar, it prevents the results materialising
[15:16:52] <Beetstra>	 Now this is curious .. the job is dead, but I still have the process running on tools-exec-04 .. twice ..
[15:17:02] <Beetstra>	 under my name .. can't kill them
[15:20:21] <scfc_de>	 Processes (2998, 9313, 24598) are "DN", something IO?
[15:21:30] <scfc_de>	 "/home/beetstra/unblockbot/unblockbot.sh: No such file or directory"
[15:22:31] <Beetstra>	 now, that is solved now .. but .. 
[15:24:43] <Beetstra>	 I can not kill those processes .. 
[15:25:44] <Beetstra>	 It also looks like I have three terminal sessions there .. I have only one open now
[15:25:54] <Beetstra>	 (2 just crashed on me)
[15:26:04] <scfc_de>	 What signal are you killing them with?
[15:26:21] <Beetstra>	 -9
[15:26:29] <Beetstra>	 standard kill does not help either
[15:28:02] <Beetstra>	 kill -9 -1 leaves a handful
[15:28:37] <scfc_de>	 Probably due to the IO hanging ("DN").  (You don't use pagecounts, do you?)
[15:29:50] <Beetstra>	 eh, don't think so
[15:30:06] <Beetstra>	 the real bot is not running, it is the start-script that seems to be the only thing hanging
[15:31:26] <Beetstra>	 and another hanging terminal
[15:31:28] <Beetstra>	 sigh
[15:33:47] <Beetstra>	 Is there any other way of really killing all my processes on tools-exec-04
[15:34:25] <mutante>	 killall -u username 
[15:34:31] <Beetstra>	 OK, lets try
[15:34:54] <scfc_de>	 mutante: The processes are "DN", with PID1 being parent.
[15:35:11] <Beetstra>	 nope, killed my active terminal, the rest is still alive after logging in again
[15:35:46] <scfc_de>	 mutante: Any good idea how to clean them up?
[15:36:47] <mutante>	 hmmm.. not really
[15:37:02] <mutante>	 tried to lookup a page which is broken itself :p
[15:38:16] <scfc_de>	 Beetstra: How relevant are the hanging processes for you (apart from a nuisance :-))?
[15:39:59] <scfc_de>	 Coren: If you got some time, +2s on https://gerrit.wikimedia.org/r/{66264,84288,100147,102630,105453,105192} would be appreciated :-).
[15:40:04] <mutante>	 "Generally, send 15, and wait a second or two, and if that doesn't work, send 2, and if that doesn't work, send 1. If that doesn't, REMOVE THE BINARY because the program is badly behaved! "
[15:40:09] <mutante>	 that was the quote i was looking for
[15:41:01] <Beetstra>	 Hmm, I could try and start the bot and see if it runs .. and just ignore these.  Did not think of that
[15:41:18] <scfc_de>	 mutante: Beetstra already tried -9, and he removed the script :-).  Removing bash and perl is not an option, I'm afraid.
[15:41:38] <mutante>	 scfc_de: ok, well the second part of that is "Don't use kill -9. Don't bring out the combine harvester just to tidy up the flower pot. " but yea:)
[15:41:55] <mutante>	 hmmm
[15:42:15] <scfc_de>	 Beetstra: BTW, if in any way possible, you should run your bot under a tool account, even if you are the sole maintainer.
[15:42:52] <Beetstra>	 I know, should work on that
[15:43:01] <anomie>	 Coren: Is there a difference between the "task" and "continuous" queues, besides jsub/jstart? I'm just curious.
[15:43:26] <Coren>	 anomie: Some settings differ; including (most importantly) a different restart scheme.
[16:04:12] <scfc_de>	 Coren: Thanks.
[16:06:55] <anomie>	 Wow, load on -04 is 253.5%
[16:06:56] <scfc_de>	 andrewbogott: Giving users shell access or adding them to the Tools project always takes quite some time (half a minute?).  Is there some way to speed it up?
[16:07:36] <andrewbogott>	 scfc_de: You mean the page load after you make the change?
[16:08:06] <andrewbogott>	 I'm not sure why it's so slow.  Plenty of other ldap changes are much faster.
[16:09:27] <scfc_de>	 andrewbogott: I mean page "processing", i. e. https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addmember&projectname=tools&member=andrewbogott (with a non-member as a user :-)) and then hitting "Submit".
[16:09:46] <scfc_de>	 (And the same for the special page for user rights management.)
[16:09:57] <andrewbogott>	 Ryan just fixed a bunch of slow page loads.  That one is still slow?
[16:10:06] <scfc_de>	 Was some hours ago.
[16:11:19] <andrewbogott>	 hm.  I don't have any immediate ideas -- you can file a bug and someone may look at it :/
[16:14:41] <scfc_de>	 It's not that important; it's just odd because I can't imagine *what* makes the LDAP update slow.  There shouldn't be that many writing accesses to LDAP, I believe.
[16:17:12] <andrewbogott>	 I agree -- I don't know why it's slow.
[16:17:54] <hedonil>	 Coren: scfc_de: btw (minor minor): the anchor-tags  like https://tools.wmflabs.org/#pywikibot are broken with the new design.
[16:19:13] <mutante>	 "Your Toolserver account expires tomorrow" .. heh, i think that's fine:)
[16:24:16] <scfc_de>	 hedonil: Where are those links coming from?
[16:24:51] <hedonil>	 scfc_de: eg. from https://tools.wmflabs.org/?status
[16:25:15] <Coren>	 hedonil: jorm is the one who did the revamp.
[16:27:06] <hedonil>	 Coren: the design is pretty cool, but iirc  design = form + functionality  ;)
[16:27:23] <scfc_de>	 hedonil: Thanks, I'll fix it later.
[16:28:00] <hedonil>	 scfc_de: can't live without that :-D
[16:28:36] <grrrit-wm>	 (03PS5) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[16:34:31] <grrrit-wm>	 (03PS6) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[16:35:50] <hedonil>	 scfc_de: to make these tables sortable in a second, I've adapted this tiny JS - I think I go it from wikipedia 
[16:35:53] <hedonil>	 scfc_de: https://tools.wmflabs.org/wikiviewstats/dev/sortable.js
[16:36:51] <scfc_de>	 hedonil: jorm already made all tables sortable?!
[16:37:27] <hedonil>	 scfc_de: not for all columns
[16:39:30] <scfc_de>	 hedonil: Which not?
[16:39:48] <hedonil>	 scfc_de: oh, I have to correct this, all columns are sortable ;)
[16:41:19] <hedonil>	 scfc_de: but with the maintainers column there's still the issue with multiple maintainers
[16:42:33] <grrrit-wm>	 (03PS7) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[16:49:01] <grrrit-wm>	 (03PS8) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[16:51:04] <aude>	 Coren or andrewbogott ping
[16:51:20] <andrewbogott>	 aude, what's up?
[16:51:59] <aude>	 in project storage for wikidata, ordinary users cannot create directories
[16:52:11] <aude>	 i have to sudo as root, which seems wrong
[16:52:20] <aude>	 how is it supposed to work?
[16:54:24] <andrewbogott>	 I don't think it's wrong… all of your users have sudo don't they?
[16:54:33] <aude>	 we do
[16:55:25] <andrewbogott>	 So… /data/project is mounted like any other volume.  Seems normal to me.
[16:55:28] <aude>	 that works though on toolserver afaik i could create shared directories without being root
[16:55:36] <aude>	 ok
[16:55:58] <andrewbogott>	 Hm, I don't know about toolserver, maybe that's customized somehow.
[16:56:14] <andrewbogott>	 Probably you could just change the permissions on that whole volume if it suits you :)
[16:56:32] <aude>	 not sure how to do since it is shared across instances
[16:57:06] <grrrit-wm>	 (03PS9) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[16:58:53] <aude>	 okay for now, i supposed
[16:58:55] <aude>	 suppose*
[17:12:26] <grrrit-wm>	 (03PS10) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[18:06:48] <Tanvir>	 Hello, when I set a job with jsub it turns out it's running once in an hour. Any idea how can I make this job repeatitive to be done every 10 minutes?
[18:11:41] <hedonil>	 Tanvir: ~$ crontab -e
[18:12:37] <hedonil>	 Tanvir: */10  * * **  jsub <your job>
[18:13:08] <Tanvir>	 Okay, I am trying.
[18:14:04] <mutante>	 export VISUAL=editoryoulike
[18:14:18] <mutante>	 optional, before crontab -e 
[18:14:40] <Tanvir>	 That I already did mutante.
[18:15:10] <Tanvir>	 Tried that, testing now. Hope it will work.
[18:15:27] <Tanvir>	 Anyway, how do I get rid of the .err and .out files?
[18:16:04] <hedonil>	 Tanvir: hehe, another cron job  rm ;)
[18:16:06] <giftpflanze>	 -j y -o /dev/null ?
[18:16:11] <mutante>	 redirect stdout and stderr to /dev/null ?
[18:16:30] <mutante>	 in your existing crontab command
[18:16:47] <mutante>	 but no debugging then :)
[18:16:52] <Tanvir>	 hedonil, excuse me? I don't understand you.
[18:17:29] <Tanvir>	 mutante, you mean, I need to use -stderr when using jsub?
[18:17:41] <mutante>	 cron jobs that output stuff create mails , so called cronspam
[18:17:51] <mutante>	 so they should always make sure to be silent
[18:17:58] <mutante>	 and just log to files ..or nowhere
[18:18:10] <Tanvir>	 I remove cron reports with > /dev/null >2&1
[18:18:20] <mutante>	 Tanvir: i dunno know about jsub specifically
[18:18:30] <mutante>	 that is just about cron 
[18:18:45] <mutante>	 Tanvir: sounds good the
[18:18:47] <mutante>	 n
[18:19:19] <giftpflanze>	 with jsub you can use -quiet
[18:19:23] <Tanvir>	 But I mean about the .err and .out files I see in my home as a byproduct of the jobs I run.
[18:19:44] <giftpflanze>	 Tanvir: 18:16 < giftpflanze> -j y -o /dev/null
[18:19:46] <Tanvir>	 giftpflanze, that's what I thought and also used, but I still see the .err files.
[18:20:03] <giftpflanze>	 the -quiet is only about cron mail
[18:20:21] <hedonil>	 Tanvir: if it can't be suppressed, just add another cron job that rm's the .err and .out files
[18:20:40] <Tanvir>	 hedonil, how? Could you tell me please?
[18:20:42] <giftpflanze>	 Tanvir: you used -j y -o /dev/null and have no .out file but a .err file?
[18:20:53] <Tanvir>	 No, that one I didn't use.
[18:21:00] <Tanvir>	 I will try that, giftpflanze.
[18:21:25] <Tanvir>	 So you mean, I need to use "-j y -o /dev/null" in the string?
[18:21:41] <giftpflanze>	 as an option to jsub
[18:21:47] <Tanvir>	 Okay.
[18:21:58] <Tanvir>	 Will try that. Thanks, giftpflanze.
[18:22:23] <giftpflanze>	 np
[18:29:55] <hedonil>	 Tanvir: e.g:  */15 * * * * rm /data/project/<yourtoolname>/*.err
[18:34:51] <hedonil>	 Tanvir: but, to my mind these files don't hurt. If you jsub your job with -N <anyname>. All messages will always be appended to <anyname>.err
[18:46:56] <andrewbogott>	 legoktm: you around?  I could use a hand understanding your patch 'Replace usage of SpecialPage::getTitle with getPageTitle'
[18:47:47] <chrismcmahon>	 any reason texvc might have been removed or corrupted on beta labs?  https://bugzilla.wikimedia.org/show_bug.cgi?id=59777
[18:51:58] <chrismcmahon>	 thanks Reedy
[18:59:15] <scfc_de>	 !log tools tools-login, tools-mail: rm -f /var/log/exim4/paniclog (apparently some artifacts of the LDAP failure)
[18:59:17] <labs-morebots>	 Logged the message, Master
[20:08:51] <andrewbogott>	 Coren (or anyone), can I get some mediawiki admin help?
[20:09:05] <Coren>	 Sure.  What be up?
[20:09:29] <andrewbogott>	 When I update the mediawiki version on wikitech-test, everything breaks.
[20:09:54] <andrewbogott>	 https://wikitech-test.wmflabs.org/wiki/
[20:10:11] <andrewbogott>	 I don't know if that's because of fancy stuff Ryan has been doing, or if I need to do an extra upgrade step or...what.
[20:11:06] <Reedy>	 Find out what the database query is! :)
[20:11:54] <Coren>	 andrewbogott: did you run the maintenance/update.php?
[20:12:04] <andrewbogott>	 hm, nope.  Let me do that...
[20:12:46] <andrewbogott>	 Ah, well, that seems to have fixed it :)
[20:13:09] <andrewbogott>	 While we're on the topic… do you happen to know how I can switch from HEAD to the latest wmf release branch?
[20:13:16] <andrewbogott>	 I've done that before but struggling with syntax atm
[20:15:04] <Coren>	 git is my favorite enemy.  I always have to google everything beyond the normal commit-review-pull cycle.  :-)
[20:15:41] <andrewbogott>	 I feel like it should just be git checkout -b wmf/1.23wmf3 wmf/1.23wmf3
[20:18:36] <andrewbogott>	 Ah!  git checkout -b wmf/1.23wmf3 origin/wmf/1.23wmf3
[20:18:59] <Reedy>	 WE're upto wmf9 though now ;)
[20:25:45] <hoo>	 I guess you people are aware: https://wikitech.wikimedia.org/wiki/
[20:25:57] <hoo>	 andrewbogott: petan: ^
[20:26:11] <andrewbogott>	 hoo, I'm upgrading, will be broken for just a minute or two
[20:26:18] <hoo>	 ah, fine :)
[20:26:53] <andrewbogott>	 Or would, if anyone had done a submodule update in the last year :(
[20:27:20] * andrewbogott  taps foot impatiently

[20:27:50] <andrewbogott>	 ok, done
[20:36:54] <andrewbogott>	 labs-morebots: you ok?
[20:36:54] <labs-morebots>	 I am a logbot running on tools-exec-02.
[20:36:55] <labs-morebots>	 Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[20:36:55] <labs-morebots>	 To log a message, type !log <msg>.
[20:36:57] <andrewbogott>	 hm
[20:57:28] <grrrit-wm>	 (03PS11) 10Tim Landscheidt: WIP: Fix build and run-time dependencies [labs/toollabs] - 10https://gerrit.wikimedia.org/r/104951 
[20:57:47] <Cyberpower678>	 @seen petan 
[20:57:48] <wm-bot>	 Cyberpower678: I have never seen petan 
[20:58:06] <Cyberpower678>	 Lovely
[20:58:28] <Cyberpower678>	 Anybody know where I can find petan?
[20:59:44] <Cyberpower678>	 petan, ping
[20:59:54] <scfc_de>	 Cyberpower678: On wiki?  By mail?  Wait for when he returns?
[21:00:17] <Cyberpower678>	 Here?  He appears to be online, now.
[23:15:15] <giftpflanze>	 coren: may ipester you again with https://bugzilla.wikimedia.org/show_bug.cgi?id=58949
[23:15:52] <Coren>	 giftpflanze: In progress