[09:12:57] <gtirloni>	 !log tools.admin restarted service (T213147)
[09:13:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL
[09:13:00] <stashbot>	 T213147: Toolforge Home Page is CRITICAL - https://phabricator.wikimedia.org/T213147
[09:59:44] <gtirloni>	 !log tools rebooted tools-checker-01 (T213252)
[09:59:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:59:48] <stashbot>	 T213252: toolscheckerctl fails to start checks - https://phabricator.wikimedia.org/T213252
[14:42:04] <andrewbogott>	 !log tools experimentally moving tools-paws-worker-1019 to eqiad1
[14:42:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:27:22] <fuzheado>	 https://paws.wmflabs.org/ -> 502 Bad Gateway
[16:27:36] <andrewbogott>	 fuzheado: that's my fault, we're working on it
[16:27:50] <fuzheado>	 Cool, thanks!
[16:33:12] <doctaxon>	 Hi! Are there problems with the enwiki databases?
[16:33:59] <doctaxon>	 select count(*) from revision join page on rev_page = page_id where p
[16:34:34] <doctaxon>	 age_title = 'Castle_Conway' and page_namespace = 0
[16:34:47] <doctaxon>	 SQL queries like this ^ do not work 
[16:35:36] <doctaxon>	 chasemp andrewbogott anyone here?
[16:47:29] <bstorm_>	 in enwiki?  checking
[16:48:31] <bstorm_>	 There's a problem with s5 replication, but I'm not aware of much on enwiki...looking around
[16:51:02] <bstorm_>	 Definitely looking slow...
[16:52:34] <bstorm_>	 Ah I see it
[16:53:31] <bstorm_>	 doctaxon: looks like someone dropped a column in the page table.  I'll check if they are rebuilding the views yet or if that is something I can jump on.
[16:56:25] <doctaxon>	 bstorm_ jynus: is it running now
[16:56:36] <doctaxon>	 now it's okay
[16:56:42] <bstorm_>	 Ah ok :)
[16:56:52] <bstorm_>	 Whatever it was, I think someone was in the process of resolving it
[16:57:30] <chasemp>	 bstorm_: also being talked about in -databases channel fyi
[16:57:43] <bstorm_>	 There's a databases channel?!?!?!
[16:57:55] <bstorm_>	 :)
[16:58:07] <gtirloni>	 T212254
[16:58:07] <stashbot>	 T212254: Drop valid_tag table - https://phabricator.wikimedia.org/T212254
[16:58:57] * bstorm_ joined
[17:00:46] <doctaxon>	 ya, thank you, it is working fine now
[17:12:15] <revi>	 the new bigbrother doesn't work with symlinks, right?
[17:17:01] <andrewbogott>	 !log tools moving paws-worker-1017 and paws-worker-1016 to eqiad1
[17:17:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:17:59] <gtirloni>	 revi: it submits the command just as you'd do manually. i'd suggest using the full path to your command just in case
[17:18:10] <andrewbogott>	 fuzheado: paws should be back now :)
[17:18:23] <revi>	 seems someone else from our team fixed it
[17:18:33] <fuzheado>	 andrewbogott: thanks much!
[17:18:42] <revi>	 just wondering if jlocal is something different from jsub or qsub like thing?
[17:18:49] <andrewbogott>	 fuzheado: gtirloni and chicocvenancio fixed it, I just broke it :)
[17:18:52] <revi>	 so I can just put -quiet on bigbrother jobs
[17:20:13] <Hauskatze>	 !log tools.stewardbots Disable jlocal crontab commands
[17:20:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[17:20:30] <gtirloni>	 revi: you could edit bigbrother.sh and add -quiet there, the bigbrother.sh is totally yours :)
[17:21:16] <revi>	 lemme see…
[17:21:29] <revi>	 this is first time I ssh to box after the bigbrother migration
[17:22:11] <revi>	 I actually thought it was something shared lol
[17:22:51] <revi>	 Hauskatze: lol you do that while I was trying to fix spambot
[17:23:45] <bd808>	 revi: jlocal is for very short lived processes that actually run directly on the cron instance rather than on the grid. Its needed for the bigbrother.sh script because the normal grid nodes are not marked as job submission hosts, so a job launched on the grid would not be able to start the monitored job if it is not found
[17:25:26] <revi>	 I guess submit_job function is where I need to put
[17:25:29] <revi>	 -quiet?
[17:25:34] <revi>	 (mobile sucks)
[17:27:07] <gtirloni>	 revi: yes
[17:27:18] <revi>	 thanks!
[17:28:26] <revi>	 !log tools.stewardbots fix bigbrother.sh to do their job quietly & re-enable the bigbrother via crontab
[17:28:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[17:29:45] <revi>	 thanks gtirloni and bd808!
[17:29:52] <revi>	 probably fixed by that
[17:30:36] <gtirloni>	 nice!
[17:31:12] <Hauskatze>	 revi: oops, but glad you fixed it -- methinks we lack coordination :)
[17:31:39] <revi>	 well that was fine in case there was floodquit while I was fixing it
[17:31:58] <revi>	 that would've prevented my inbox being spammed every 5 minutes again
[17:32:21] * Hauskatze doesn't have the email open, floods revi's
[17:33:01] <revi>	 >_<
[17:33:12] <revi>	 RIP my mailbox
[17:45:59] <revi>	 didn't work quite well
[17:46:02] <revi>	 > Usage: /data/project/stewardbots/bigbrother.sh <jobname> <command>
[17:49:06] <revi>	 !log tools.stewardbots disable bigbrother again
[17:49:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[17:51:10] <revi>	 can someone (maybe Hauskatz) make sure crontab is gone? I got a crontab mail
[17:51:20] <revi>	 I need to sleep so
[17:51:47] <gtirloni>	 revi: what kind of emails are you receiving? is bigbrother.sh restarting your jobs constantly?
[17:51:58] <revi>	 looks like so
[17:52:15] <revi>	 this is what I got https://usercontent.irccloud-cdn.com/file/TdI9rkq8/IMG_7182.PNG
[17:53:05] <gtirloni>	 "/usr/bin/python /data/project/stewardbots/StewardBot/StewardBot.py"
[17:53:08] <gtirloni>	 this needs to be a single command
[17:53:11] <gtirloni>	 was it like that before?
[17:53:17] <revi>	 IIRC it was
[17:53:48] <revi>	 or it was somehow converted to that while bigbrother migration?
[17:54:07] <gtirloni>	 https://www.irccloud.com/pastebin/PwGGxHuz/
[17:54:10] <gtirloni>	 this should work now
[17:54:32] <revi>	 thx
[17:54:37] <revi>	 rxy: ^ in case
[17:54:52] <revi>	 (you're still working on this)
[17:54:57] <revi>	 anyway 3am wow
[17:54:58] <revi>	 nini
[17:55:07] <gtirloni>	 g'night :)
[17:55:11] <rxy>	 g'night
[17:55:22] <revi>	 (tho I recall saying this lol)
[18:34:46] <addshore>	 gtirloni: im getting your test emails :P
[18:42:59] <gtirloni>	 addshore: thanks :) i was testing the wrong thing lol, sorry
[18:52:14] <rxy>	 gtirloni: hi, around?
[18:53:16] <rxy>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Bigbrother_(Deprecated)  isn't work for my python script.
[19:00:23] <gtirloni>	 rxy: what's your tool? what's the error?
[19:00:42] <rxy>	 at tools.stewardbots
[19:00:47] <rxy>	 no error output
[19:01:11] <rxy>	 but it seems can't found python env
[19:02:38] <gtirloni>	 how do you usually start it?
[19:03:16] <rxy>	 we usually using ` /data/project/stewardbots/StewardBot/restart_stewardbot.sh `
[19:04:13] <gtirloni>	 looks like there's another wrapper script that talks directly with the grid
[19:04:16] <gtirloni>	 the old bigbrother had this:
[19:04:18] <gtirloni>	 #jstart -N stewardbot -mem 2G python /data/project/stewardbots/StewardBot/StewardBot.py
[19:04:18] <gtirloni>	 #jstart -N sulwatcher -mem 2G python /data/project/stewardbots/SULWatcher/SULWatcher.py
[19:05:16] <rxy>	 I guess "jlocal" is system user without any path?
[19:05:18] <gtirloni>	 -quiet is not a valid option for jstart
[19:05:50] <gtirloni>	 i've removed it from bigbrother.sh
[19:07:30] <rxy>	 thx
[19:07:44] <gtirloni>	 the bot still can't start though, even manually without using bigbrother.sh
[19:08:53] <gtirloni>	 ah ok, /mnt/nfs/labstore-secondary-tools-project/stewardbots/public_html/StewardBot/StewardBot.py wasn't executable (`chmod +x`)
[19:08:57] <gtirloni>	 should be fine now
[19:10:12] <rxy>	 ah, thanks
[19:16:59] <rxy>	 work correctly! Thanks a lot :)
[19:24:52] <gtirloni>	 :)
[20:16:56] <andrewbogott>	 !log tools moving tools-paws-worker-1013 and tools-paws-worker-1007 to eqiad1
[20:16:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:21:08] <bd808>	 !log admin neutron quota-update --tenant-id tools --port 256
[22:21:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL