[00:21:39] <bd808>	 !log tools.stashbot Crossing fingers and moving bot and webservice to new k8s cluster
[00:21:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[00:37:28] <Zoranzoki21>	 !help Hi, I want to run this https://pastebin.com/GzpGBkTz each friday on my srwiki-dev instance. How I can make it possible?
[00:37:28] <wm-bot>	 Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[00:38:10] <bd808>	 Zoranzoki21: crontab -e ?
[00:39:39] <Zoranzoki21>	 What?
[00:39:53] <Zoranzoki21>	 I don't understand your reply
[00:40:42] <bd808>	 you want to run a periodic repeating job. That is what the cron service is designed to do. So make a shell script and then tell cron to run it
[00:41:09] <bd808>	 `crontab -e` is the command to edit your account's cron configuration
[00:42:09] <Zoranzoki21>	 # HEADER: This file was autogenerated at 2019-11-09 18:12:10 +0000 by puppet.
[00:42:10] <Zoranzoki21>	 # HEADER: While it can still be managed manually, it is definitely not recommended.
[00:42:12] <Zoranzoki21>	 # HEADER: Note particularly that the comments starting with 'Puppet Name' should
[00:42:13] <Zoranzoki21>	 # HEADER: not be deleted, as doing so could cause duplicate cron jobs.
[00:42:15] <Zoranzoki21>	 # Puppet Name: prometheus_ssh_open_sessions
[00:42:19] <Zoranzoki21>	 ?
[00:42:49] <bd808>	 I would not recommend running git commands as root
[00:43:17] <Zoranzoki21>	 var/www/html no allows me to open as non-root user
[00:44:26] <Zoranzoki21>	 I mean: /var/www/html allows me reading, no writing
[00:44:31] <bd808>	 you are root so you can change permissions on files and directories. But if you want to do all of your management as root that is your decision.
[00:46:39] <bd808>	 Those warnings in the root user's crontab are real, but like they say you can still manually add things to that configuration. Puppet should not remove new lines, but it should revert changes to likes that it knows to manage
[00:46:42] <Zoranzoki21>	 zoranzoki21@srwiki-dev:/var/www/html$ sudo chown -R zoranzoki21:zoranzoki21 /var/www/html
[00:46:43] <Zoranzoki21>	 chown: invalid group: 'zoranzoki21:zoranzoki21'
[00:47:11] <bd808>	 we do not make per user groups in the Cloud VPS shared auth system (LDAP)
[00:47:33] <bd808>	 type `groups` to see what groups your user is a member of
[00:48:01] <Zoranzoki21>	 zoranzoki21@srwiki-dev:/var/www/html$ groups
[00:48:02] <Zoranzoki21>	 wikidev project-bastion project-tools tools.testwiki tools.discordwiki tools.vagrant2 project-srwiki-dev tools.srwiki-dev
[00:48:04] <Zoranzoki21>	 zoranzoki21@srwiki-dev:/var/www/html$
[00:50:56] <Zoranzoki21>	 ------
[00:51:01] <Zoranzoki21>	 zoranzoki21@srwiki-dev:/var/www/html$ ls -ld /var/www/html
[00:51:02] <Zoranzoki21>	 drwxr-xr-x 18 root root 4096 Dec 22 00:41 /var/www/html
[00:51:18] <bd808>	 the "wikidev" or "project-srwiki-dev" groups would be reasonable to use
[00:52:28] <Zoranzoki21>	 Shows invalid user error
[00:53:00] <Zoranzoki21>	 Ok, I used zoranzoki21 and now works
[00:54:46] <bd808>	 If you are unfamiliar with cron, https://help.dreamhost.com/hc/en-us/articles/215767047-Creating-a-custom-Cron-Job is a detailed tutorial
[01:00:14] <Zoranzoki21>	 I am a bit familiar with cron
[01:00:26] <Zoranzoki21>	 Thank you for help, I created cron jobs :)
[01:00:40] <Zoranzoki21>	 For cron I just needed how to format date/time and I found it here https://crontab.guru/#0_20_*_*_THU
[01:01:24] <Zoranzoki21>	 This should be ok: https://prnt.sc/qe5cal
[01:01:26] <Zoranzoki21>	 :)
[01:02:08] <bd808>	 nice
[01:02:59] <Zoranzoki21>	 Thank you, I'm going to do this and for another instance now :)
[01:30:55] <Zoranzoki21>	 Although I get nervous sometimes, I can learn a lot here :)
[01:31:41] <Zoranzoki21>	 I learned how to create a cron to send notification on my email about successful execution of script :)
[01:45:53] <Zoranzoki21>	 All works! thanks again bd808 and sorry for spam :P
[01:46:21] <bd808>	 Zoranzoki21: no worries. I'm glad you learned how to make it work
[01:53:28] <Zoranzoki21>	 I have another question :D
[01:53:51] <Zoranzoki21>	 I can't see Serbian letters like č ć š đ ž in terminal when I am connected on instance
[01:54:03] <Zoranzoki21>	 It shows empty space
[01:54:34] <Zoranzoki21>	 So cron sends command to tools.wmflabs instance without these letters
[01:54:47] <Zoranzoki21>	 How I can fix this?
[01:55:08] <Zoranzoki21>	 In local I can see it normally
[01:56:13] <bd808>	 hmmm... I would think the default locale is utf8, but maybe it is not?
[01:56:25] <Zoranzoki21>	 I don't know
[01:56:33] <Zoranzoki21>	 Serbian letters works with utf8
[01:57:47] <bd808>	 Zoranzoki21: what does `locale` on the remote server tell you? It will output a bunch of lines, so you should use a pastebin to share the output
[01:58:11] <bd808>	 I would expect it to show LANG=en_US.UTF-8 (and other things)
[01:58:29] <Zoranzoki21>	 https://pastebin.com/wXP04PQQ
[02:00:26] <Zoranzoki21>	 It is for srwiki-dev.srwiki-dev
[02:00:30] <Zoranzoki21>	 For discordwiki.srwiki-dev: https://pastebin.com/wSBgntTa
[02:01:28] <bd808>	 ok. I think the "locale: Cannot set LC_ALL to default locale: No such file or directory" line is telling us that your ssh client really wants to set the local to sr_RS to match your local locale.
[02:01:48] <Zoranzoki21>	 I use default application on Linux Kubuntu for connecting via ssh
[02:02:03] <Zoranzoki21>	 It's Konsole
[02:03:27] <Zoranzoki21>	 And my system is on English too
[02:03:55] <Zoranzoki21>	 Output of command on my local system: https://pastebin.com/JXis0BEA
[02:05:15] <bd808>	 Lets try generating the sr_RS locale and see if that makes things different.
[02:05:42] <bd808>	 this tutorial looks like it might help -- https://wiki.bicomsystems.com/HOWTO_Linux_Locale
[02:06:40] <bd808>	 basically, edit /etc/locale.gen to uncomment the "sr_RS UTF-8" line, and then run `sudo locale-gen` 
[02:08:41] <Zoranzoki21>	 I can see Serbian letters now
[02:10:05] <bd808>	 \o/
[02:11:00] <Zoranzoki21>	 And now I can see mail with these letters
[02:11:02] <Zoranzoki21>	 :)
[02:13:04] <Zoranzoki21>	 And I no see  "locale: Cannot set LC_ALL to default locale: No such file or directory" anymore
[02:21:01] <Zoranzoki21>	 I'm going to sleep, good night!
[04:45:57] <bd808>	 !log tools.bash Migrated to PHP 7.3 and new kubernetes cluster. JS and CSS broken due to T241310
[04:45:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bash/SAL
[04:46:00] <stashbot>	 T241310: Kubernetes ingress passes it's port & proto to apps rather than the port & proto from the front proxy - https://phabricator.wikimedia.org/T241310
[09:37:56] <arturo>	 !log admin cloudvirt1013 is down for good. Apparently powered off. I can't even reach it via iLO
[09:37:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:45:10] <arturo>	 !log admin cloudvirt1013 is back (did it alone) T241313
[09:45:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:45:12] <stashbot>	 T241313: cloudvirt1013: server down for no reason (power issue?) - https://phabricator.wikimedia.org/T241313
[12:29:34] <Amir1>	 !log ladsgroup@codesearch4:~$ sudo service hound-armchairgm restart (T241319)
[12:29:35] <stashbot>	 Amir1: Unknown project "ladsgroup@codesearch4:~$"
[12:29:35] <stashbot>	 T241319: ArmchairGM repo indexing broken entirely - https://phabricator.wikimedia.org/T241319
[12:29:50] <Amir1>	 !log codesearch ladsgroup@codesearch4:~$ sudo service hound-armchairgm restart (T241319)
[12:29:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[18:25:56] <Zoranzoki21>	 !help Hi, how I can create snapshot of instnace?
[18:25:56] <wm-bot>	 Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[18:26:15] <bd808>	 Zoranzoki21: we do not support snapshots currently
[18:27:01] <Zoranzoki21>	 I created T240417 last week, but I see snapshot option, and I wanted just to ask because I see option
[18:27:01] <stashbot>	 T240417: Add option for cloning instances on horizon - https://phabricator.wikimedia.org/T240417
[18:27:24] <Zoranzoki21>	 On creating instance to select snapshot, but I do not see option for creating snapshot
[18:27:52] <bd808>	 we do not have a proper storage layer right now for snapshots
[18:27:59] <Zoranzoki21>	 I understand, ok then
[18:52:37] <bd808>	 !log tools Disabled Puppet on tools-proxy-06.tools.eqiad.wmflabs to test nginx config change (T241310)
[18:52:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:52:41] <stashbot>	 T241310: Kubernetes ingress passes it's port & proto to apps rather than the port & proto from the front proxy - https://phabricator.wikimedia.org/T241310
[20:13:50] <bd808>	 !log tools Enabled Puppet on tools-proxy-06.tools.eqiad.wmflabs after nginx config test (T241310)
[20:13:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:13:53] <stashbot>	 T241310: Kubernetes ingress passes it's port & proto to apps rather than the port & proto from the front proxy - https://phabricator.wikimedia.org/T241310
[22:29:39] * Krinkle <3 bd808  for making "tool-fourohfour", e.g. https://tools.wmflabs.org/cover-me
[22:30:22] <hauskatze>	 heh
[22:31:03] <hauskatze>	 Krinkle: is a production equivalent of 'deployment-cpjobqueue' and what's suposed to be done there?
[22:31:10] <hauskatze>	 server I mean
[22:31:14] <hauskatze>	 on eqiad.wmnet
[22:31:28] <hauskatze>	 I keep getting EMERG errors for beta cluster
[23:48:40] <andrewbogott>	 !log admin restarting nova-conductor and nova-api on cloudcontrol1003 and 1004
[23:48:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL