[08:07:16] <Majavah>	 !log toolsbeta creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / T153810
[08:07:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[08:07:19] <stashbot>	 T153810: Make switching Redis server simpler - https://phabricator.wikimedia.org/T153810
[08:44:08] <billinghurst>	 Is anyone able to tell me who/how/when pywikibot is rolled out to toolserver?  I have told that scripts have got ahead of the program "Looks like Pywikibot is 6.1.0 but archivebot is 6.2.0.dev0. Are scripts provided in a different way?" Thanks.
[08:44:19] <billinghurst>	 *have been
[09:00:21] <arturo>	 billinghurst: that would be us if you use the grid, and probably yourself if you use kubernetes (because the venv?)
[09:00:42] <arturo>	 or... you can use the python venv in the grid too, no?
[09:00:56] <arturo>	 so: it depends!
[09:01:46] <legoktm>	 billinghurst: in theory it's just updated every night
[09:01:47] <billinghurst>	 arturo, so why would we have the the scripts out of sync?  or how do I check whether they are out of sync?
[09:01:58] <legoktm>	 which tool is this for?
[09:02:09] <billinghurst>	 pywikibot legoktm 
[09:02:27] <legoktm>	 no, I meant which tool are you running pywikibot from?
[09:02:43] <legoktm>	 when you run "become <tool>"
[09:02:45] <billinghurst>	 archivebot.py
[09:02:58] <billinghurst>	 https://phabricator.wikimedia.org/T282766
[09:03:11] <billinghurst>	 oh wikisource-bot
[09:04:10] * legoktm takes a peek
[09:05:53] <legoktm>	 ok, I see the problem
[09:06:57] <legoktm>	 the short version is you should run $ python3 /shared/pywikibot/core_stable/scripts/archivebot.py
[09:07:31] <legoktm>	 for some reason /shared/pywikibot and /shared/pywikipedia are set up differently
[09:07:44] <legoktm>	 I'll write a longer answer on the task
[09:07:46] <legoktm>	 billinghurst: ^^
[09:07:51] <billinghurst>	 thanks
[09:08:02] <billinghurst>	 you continue to be a star :-)
[09:08:41] <legoktm>	 :))) happy to help
[09:10:01] <addshore>	 Hey hey! I just noticed the dashboard at https://grafana-labs.wikimedia.org/d/000000590/instance-details is gone, is there a replacement?
[09:15:55] <addshore>	 I seem to have found cloud-vps-project-board but it doesnt have the "memory compaction" panel I was looking at before
[09:20:16] <billinghurst>	 okay, they have updated the pywikibot information at https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Pywikibot& about paths since I did mine
[09:22:07] <legoktm>	 I think things were switched around when "stable" was introduced
[09:27:51] <arturo>	 addshore: that's probably the result of us failovering cloudmetrics servers. We lost some grafana configs I'm afraid
[09:28:05] <addshore>	 aah right!
[09:28:19] <addshore>	 *tries to find the metric names in graphite*
[09:28:36] <arturo>	 addshore: the right thing to do is to open a phab task and figure out a mechanism to persist grafana config between servers (probably, storing them in puppet)
[13:49:02] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 95f40ac9d5 (Norwegian Bokmål masculine/neuter nouns)
[13:49:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[14:02:18] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 67e7cf3dfb (rename Swedish adjective template)
[14:02:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[15:45:41] <bstorm>	 !log admin re-running wikireplicas-dns after refactor of config to make sure it doesn't change anything
[15:45:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[20:22:16] <chicocvenancio>	 bridgebot seems down again...
[20:22:33] <mutante>	 probably because of the netsplit
[20:22:42] <mutante>	 there wa one a couple minutes ago
[20:23:00] <mutante>	 guessing it needs to reconnect
[20:23:31] <wm-bot>	 !log tools.bridgebot <bd808> Restarting because this bot is a pile of broken crap
[20:23:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[20:23:44] <mutante>	 hehe
[20:23:57] <bd808>	 chicocvenancio: hopefully it will be back soon
[20:24:15] <chicocvenancio>	 mutante: previous message was 9 hours before the restart, its back already bd808 
[20:25:06] <bd808>	 I think I really need to split the bot into multiple separate instances for the different channels/platforms
[20:26:07] <bd808>	 chicocvenancio: if you would be at all interested in co-maintainership to restart the pod when it loses connection, I would be more than happy to add you
[20:26:53] * bd808 wonders how a health check could be made to notice when the bridges stop bridging
[20:32:49] <chicocvenancio>	 bd808: I'd be fine with being added as maintainer. 
[20:33:33] <bd808>	 chicocvenancio: {{DONE}} !
[20:33:47] <chicocvenancio>	 Not hard to create an intrusive healthcheck, can't think of an non intrusive one
[20:34:00] <bd808>	 the "fix" is `kubectl get po; kubectl del po <pod id>`
[20:34:41] <bd808>	 right now there is only one pod, so I suppose I could turn that into a one-liner with some get output formatting
[20:45:43] <chicocvenancio>	 `kubectl rollout restart deploy bridgebot.bot` should do it
[21:25:15] <bstorm>	 !log admin converted the maps and scratch volumes on cloudstore1008 (standby) to drbd T224747
[21:25:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:25:19] <stashbot>	 T224747: Move maps and scratch on cloudstore1008/9 to a DRBD failover similar to labstore1004/5 - https://phabricator.wikimedia.org/T224747
[23:03:33] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 3e2ceb0513 (l10n updates)
[23:03:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL