[03:11:50] <Cyberpower678>	 !help cyberbot-exec-iabot-01 will not let me connect.
[03:11:50] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[03:12:02] <Cyberpower678>	 It's down, and I can't seem to access it.
[03:13:43] <Cyberpower678>	 $ ssh cyberbot-exec-iabot-01.eqiad.wmflabs
[03:13:43] <Cyberpower678>	 channel 0: open failed: connect failed: No route to host
[03:13:43] <Cyberpower678>	 stdio forwarding failed
[03:13:45] <Cyberpower678>	 kex_exchange_identification: Connection closed by remote host
[03:14:00] <andrewbogott>	 does it show as down on Horizon?
[03:14:24] <Cyberpower678>	 I haven't checked.
[03:14:31] <Cyberpower678>	 I forgot that it even exists.
[03:14:32] <Cyberpower678>	 :p
[03:15:26] <andrewbogott>	 you can also see the syslog there and figure out what's going on, in case it crashed or something
[03:17:01] <Cyberpower678>	 andrewbogott: it's running
[03:17:13] <Cyberpower678>	 I only issued a reboot command to it, and then it disappeared.
[03:18:05] <andrewbogott>	 ok, back up --
[03:18:16] <andrewbogott>	 you rebooted it, and then you couldn't access it after that?
[03:18:36] <Cyberpower678>	 Yeo
[03:18:43] <Cyberpower678>	 *yep
[03:19:14] <andrewbogott>	 and how long ago was that?
[03:19:21] <Cyberpower678>	 Like 15 minutes ago.
[03:19:37] <Cyberpower678>	 I wanted to flush the RAM and give IABot a reboot.
[03:19:44] <andrewbogott>	 it seems fine now
[03:19:46] <andrewbogott>	 to me at least
[03:19:50] <Cyberpower678>	 I do that every now and then.
[03:20:33] <Cyberpower678>	 Huh.  It's been giving me that no route to host error since I rebooted, until now.
[03:20:41] <Cyberpower678>	 Weird
[03:21:08] <Cyberpower678>	 I'm back in.
[03:21:17] <andrewbogott>	 great!
[03:21:24] <andrewbogott>	 You can check the syslog and see what it was doing all that time :)
[03:24:42] <Cyberpower678>	 andrewbogott: I see what look like puppet failures?
[03:25:05] <andrewbogott>	 I wouldn't think it would need to run puppet before letting you log in
[03:27:05] <Cyberpower678>	 I see a lot of cron entries, but that's normal.  Puppet-agent keeps trying to start MariaDB.
[03:27:08] <Cyberpower678>	 And failing
[03:28:52] <zhuyifei1999_>	 shall I take a look?
[03:30:36] <zhuyifei1999_>	 (wait never mind I'm not in the sudo policy)
[03:30:39] <Cyberpower678>	 zhuyifei1999_: sure.
[03:31:45] <zhuyifei1999_>	 it doesn't allow me to read the journal
[03:32:35] <Cyberpower678>	 zhuyifei1999_: I thought you're a sysadmin?
[03:33:25] <zhuyifei1999_>	 no? I can add myself to sudo policy if you don't mind. you added me to project admin like ages ago
[03:33:41] <zhuyifei1999_>	 oh hi yuvi
[03:34:26] <Cyberpower678>	 zhuyifei1999_: sure, I don't mind.  Just don't go through my hidden stash of secrets. ;P
[03:35:48] <zhuyifei1999_>	 (absolutely no idea where that even is :P)
[03:38:23] <Cyberpower678>	 Ugh.  Something is super broken.
[03:38:23] <zhuyifei1999_>	 Logs begin at Thu 2020-05-14 03:14:01 UTC <= journal was flushed right before reboot?
[03:38:41] <Cyberpower678>	 zhuyifei1999_: dunno
[03:38:43] <Cyberpower678>	 Probably
[03:39:06] <Cyberpower678>	 My DB VM is acting funny
[03:39:09] <Cyberpower678>	 IABot will not run now.
[03:39:29] <zhuyifei1999_>	 4246 lines in journal, I don't think systemd flushes journal that frequently
[03:39:33] <zhuyifei1999_>	 reading syslog
[03:41:27] <Cyberpower678>	 Error encountered when performing Introspect schema s51059__cyberbot: Cannot proceed, because event scheduler is disabled.
[03:41:42] <Cyberpower678>	 zhuyifei1999_: can you try to run IABot?
[03:41:52] <zhuyifei1999_>	 what is the command?
[03:41:57] <Cyberpower678>	 I can't get it to start.
[03:42:40] <Cyberpower678>	 zhuyifei1999_: navigate to /home/IABot/IABot and run php deadlink.php
[03:43:21] <zhuyifei1999_>	 nagf https://tools.wmflabs.org/nagf/?project=cyberbot#h_cyberbot-exec-iabot-01_cpu says everything seems to be normal until the CPU went idle a bit before reboot
[03:43:59] <zhuyifei1999_>	 https://graphite-labs.wikimedia.org/render/?title=cyberbot-exec-iabot-01+CPU+last+day&width=800&height=250&from=-1hour&hideLegend=false&uniqueLegend=true&yMax=100&target=alias(color(stacked(cyberbot.cyberbot-exec-iabot-01.cpu.total.user)%2C%22%233333bb%22)%2C%22User%22)&target=alias(color(stacked(cyberbot.cyberbot-exec-iabot-01.cpu.total.nice)%2C%22%23ffea00%22)%2C%22Nice%22)&target=alias(color(stacked(cyberbot.cyberbot-exec-i
[03:43:59] <zhuyifei1999_>	 abot-01.cpu.total.system)%2C%22%23dd0000%22)%2C%22System%22)&target=alias(color(stacked(cyberbot.cyberbot-exec-iabot-01.cpu.total.iowait)%2C%22%23ff8a60%22)%2C%22Wait+I%2FO%22)&target=alias(alpha(color(stacked(cyberbot.cyberbot-exec-iabot-01.cpu.total.idle)%2C%22%23e2e2f2%22)%2C0.4)%2C%22Idle%22)
[03:44:03] <zhuyifei1999_>	 argh
[03:44:12] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/MzCeMqF2/
[03:44:21] <zhuyifei1999_>	 this is last hour
[03:45:09] <Cyberpower678>	 Something is completely hosed.  I have no idea if it's the DB VM or the Exec VM, or bot
[03:45:11] <zhuyifei1999_>	 so the start of then it became idle is around 2:55, matches what you said in 22:19:22 <Cyberpower678> Like 15 minutes ago.
[03:45:48] <Cyberpower678>	 I can't even begin to figure out what is broken.
[03:46:00] <zhuyifei1999_>	 I pretty much only see cron during this time
[03:46:06] <zhuyifei1999_>	 let me try to run it
[03:46:25] <zhuyifei1999_>	 do I need to sudo as some user?
[03:46:37] <Cyberpower678>	 No
[03:46:58] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/Ja9pAsYX/
[03:47:19] <Cyberpower678>	 That's exactly what I'm getting.  Ever since I rebooted.
[03:47:35] <Cyberpower678>	 I think the VM is corrupted.
[03:47:45] <Cyberpower678>	 Don't know how that happened.
[03:47:54] <zhuyifei1999_>	 is the DB on the same host?
[03:47:58] <Cyberpower678>	 No
[03:48:10] <Cyberpower678>	 It's on a 300 GB disk host.
[03:48:41] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/sS78ZoT3/
[03:49:13] <Cyberpower678>	 What am I looking at?
[03:49:19] <zhuyifei1999_>	 strace
[03:49:34] <zhuyifei1999_>	 I can ssh into cyberbot-db-01
[03:49:47] <zhuyifei1999_>	 does the mysql command line work there?
[03:51:53] <Cyberpower678>	 Yes.
[03:53:15] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/SJEHV0P6/
[03:53:24] <zhuyifei1999_>	 communication looks broken
[03:53:30] <zhuyifei1999_>	 let me trace
[03:53:59] <Cyberpower678>	 zhuyifei1999_: you need to use sudo to access it
[03:54:15] <Cyberpower678>	 I don't have the account set up to work on the host.
[03:54:27] <zhuyifei1999_>	 I was pointing out that mysql should not emit 'reading initial communication packet'
[03:54:34] <Cyberpower678>	 Oh
[03:54:40] <zhuyifei1999_>	 as in, the behavior differs across hosts
[03:54:54] <Cyberpower678>	 So likely the exec vm is hosed.
[03:55:12] <Cyberpower678>	 Because DataGrip can still access the DB.
[03:55:19] <Cyberpower678>	 But it can't introspect the tables.
[03:55:44] <zhuyifei1999_>	 I'm gonna get myself strace access
[03:56:29] <Cyberpower678>	 zhuyifei1999_: I don't suppose you can just allocate me the resources for a new VM, and I just migrate over to it?
[03:56:41] <zhuyifei1999_>	 I don't have that power
[03:56:46] <Cyberpower678>	 andrewbogott: ^>
[03:56:48] <zhuyifei1999_>	 ask andrew :P
[03:58:22] <andrewbogott>	 Best to open a ticket: https://phabricator.wikimedia.org/project/profile/2880/
[03:58:46] <Cyberpower678>	 Dammit, I'm supposed to be studying, not dealing with this nonsense.  Why does this have to happen now of all times???
[04:00:06] <Cyberpower678>	 zhuyifei1999_: any luck?
[04:00:34] <zhuyifei1999_>	 found it
[04:00:37] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/XgByul9H/
[04:01:25] * zhuyifei1999_ poof
[04:01:54] <Cyberpower678>	 What?? WHY???  When did the VM suddenly get denied access to talk to the DB>
[04:02:01] <zhuyifei1999_>	 FQDN change?
[04:02:15] <zhuyifei1999_>	 'eqiad1.wikimedia.cloud' vs 'eqiad.wmflabs'
[04:02:49] <zhuyifei1999_>	 idk, do you have a whitelist somewhere?
[04:03:11] <Cyberpower678>	 zhuyifei1999_: I haven't fudged with any of that.
[04:03:24] <zhuyifei1999_>	 whitelist?
[04:03:29] <Cyberpower678>	 So I don't know.
[04:03:57] <zhuyifei1999_>	 FQDN was done sometime ago, there was an email about it but I didn't read in detail
[04:04:57] <andrewbogott>	 isn't it just db grants?
[04:05:30] <Cyberpower678>	 My DB grants appear to be fine.
[04:05:59] <Cyberpower678>	 mysqlcheck has verified the mysql tables to be OK
[04:06:43] <zhuyifei1999_>	 does grant limit FQDN?
[04:07:27] <andrewbogott>	 hm… probably not
[04:08:45] * Cyberpower678 doesn't understand why a reboot would break all of this.
[04:09:52] <zhuyifei1999_>	 it recalculates the FQDN
[04:09:56] <zhuyifei1999_>	 in the reboot
[04:10:12] <zhuyifei1999_>	 what are your grants?
[04:10:21] <zhuyifei1999_>	 https://stackoverflow.com/questions/1559955/host-xxx-xx-xxx-xxx-is-not-allowed-to-connect-to-this-mysql-server
[04:13:09] <Cyberpower678>	 select Host,User From mysql.user;
[04:13:09] <Cyberpower678>	 +-----------------------------------------------+--------+
[04:13:09] <Cyberpower678>	 | Host                                          | User   |
[04:13:11] <Cyberpower678>	 +-----------------------------------------------+--------+
[04:13:13] <Cyberpower678>	 | %.tools.eqiad.wmflabs                         | s51059 |
[04:13:15] <Cyberpower678>	 | cyberbot-exec-01.cyberbot.eqiad.wmflabs       | s51059 |
[04:13:17] <Cyberpower678>	 | cyberbot-exec-iabot-01.cyberbot.eqiad.wmflabs | s51059 |
[04:13:19] <Cyberpower678>	 | localhost                                     | root   |
[04:13:21] <Cyberpower678>	 +-----------------------------------------------+--------+
[04:13:32] <Cyberpower678>	 It's been that way since the VM was first created.
[04:13:33] <AntiComposite>	 use a pastebin next time please
[04:13:53] <Cyberpower678>	 Grrr, my client was supposed to handle that automatically.
[04:14:08] <Cyberpower678>	 Sorry
[04:17:24] <Cyberpower678>	 zhuyifei1999_: so based on the grants above, the VM should be able to connect just fine.
[04:17:40] <zhuyifei1999_>	 23:02:16 <zhuyifei1999_> 'eqiad1.wikimedia.cloud' vs 'eqiad.wmflabs'
[04:19:31] <Cyberpower678>	 zhuyifei1999_: I'll change them up, but I have my doubts.
[04:38:35] <Cyberpower678>	 zhuyifei1999_: hmmm, now I'm getting an access denied error from IABot which shouldn't be the case since I refreshed the grants and flushed the privileges.
[04:39:08] <zhuyifei1999_>	 well, at least now it's a mariadb thing
[04:47:37] <Cyberpower678>	 zhuyifei1999_: I can't seem to get past the access denied error.
[04:47:55] <zhuyifei1999_>	 what are your grants?
[04:48:16] <Cyberpower678>	 FILE
[04:48:28] <zhuyifei1999_>	 FILE?
[04:48:48] <zhuyifei1999_>	 PHP Warning:  mysqli_connect(): (HY000/1044): Access denied for user 's51059'@'cyberbot-exec-iabot-01.cyberbot.eqiad1.wikimedia.cloud' to database 's51059__cyberbot' in /home/IABot/IABot/Core/DB.php on line 586
[04:48:55] <zhuyifei1999_>	 did you grant USAGE?
[04:49:11] <Cyberpower678>	 It never needed that in the past.
[04:49:22] <Cyberpower678>	 FILE should include usage, no?
[04:50:04] <zhuyifei1999_>	 I never heard of FILE
[04:50:08] <zhuyifei1999_>	 maybe ask a DBA?
[04:50:19] <zhuyifei1999_>	 I'm not an expert as DB security
[04:50:27] <zhuyifei1999_>	 *at
[04:56:48] <Cyberpower678>	 Okay.  I got it.
[04:56:57] <Cyberpower678>	 IABot is booting up.
[05:08:12] <Cyberpower678>	 zhuyifei1999_: thank you for your help.
[05:08:19] <zhuyifei1999_>	 np
[12:08:39] <arturo>	 !log toolsbeta created toolsbeta-acme-chief-01 VM (T252762)
[12:08:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[12:08:42] <stashbot>	 T252762: tools/toolsbeta: improve acme-chief integration - https://phabricator.wikimedia.org/T252762
[12:09:20] <arturo>	 !log toolsbeta created puppet prefix toolsbeta-acme-chief in horizon (T252762)
[12:09:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[13:04:18] <Cyberpower678>	 bd808: andrewbogott: If I nuke one of my instances, will those resources be lost to me, or can I immediately recreate a new instance?
[13:06:50] <Reedy>	 You don't lose your quota
[13:07:10] <Reedy>	 It might not be "immediate", but they should be available "soon" after (when things have processed)
[13:07:11] <Cyberpower678>	 Great.  Something is borked with my VM.
[13:07:32] <Cyberpower678>	 And it's outdated anyway, so I'm going to upgrade it.
[15:02:12] <andrewbogott>	 !log admin moving all of cloudvirt100[1-9] into the 'toobusy' host aggregate.  These are slower, have spinning disks, and are due for replacement.
[15:02:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:14:28] <yuvipanda>	 is this one full cycle of machines through since I left?
[15:15:01] <andrewbogott>	 yuvipanda: Probably, almost!  We can't actually replace them for a while though
[15:15:17] <andrewbogott>	 They're gradually disintegrating so I don't want to put any new load on them
[15:15:28] <yuvipanda>	 similar to virt100[1-6] I guss
[15:15:43] <andrewbogott>	 yeah
[15:16:02] <andrewbogott>	 We have a bunch of new hardware (ceph storage nodes and new cloudvirts w/o local storage) awaiting racking
[15:16:25] <andrewbogott>	 but the DC is mostly closed to humans due to COVID so no clear timeline for when we get to stand up the new bits
[15:18:10] <yuvipanda>	 ah, right.
[15:43:06] <andrewbogott>	 yuvipanda: are you stuck in India due to travel restrictions?
[15:43:16] <yuvipanda>	 andrewbogott: yep. indefinitely for now...
[15:43:25] <yuvipanda>	 but I made it to Goa before things hit
[15:43:48] <andrewbogott>	 And are you also mostly locked down?
[15:44:13] <andrewbogott>	 I guess in a sense, if you're not allowed to leave the house it doesn't matter that much if you're stuck away from your CA friends :/
[15:44:18] <yuvipanda>	 andrewbogott: yeah
[15:44:30] <yuvipanda>	 I've thankfully gotten myself locked in with 2 good friends
[15:44:32] <yuvipanda>	 so that isn't too bad
[15:44:43] <andrewbogott>	 Oh, nice — yeah, I feel bad for folks who are shut in solo
[15:45:02] <andrewbogott>	 And society is mostly functioning OK?  No riots, groceries available, power working?
[15:47:10] <andrewbogott>	 !log admin moving cloudvirt1004 and cloudvirt1006 to the 'ceph' aggregate for T252784
[15:47:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:47:13] <stashbot>	 T252784: Remove Ceph from cloudvirt-wdqs100x, Add ceph to cloudvirt1004 and cloudvirt1006 - https://phabricator.wikimedia.org/T252784
[15:48:21] <kalle>	 Hi all. I just removed an instance and recreated it with Debian buster. I must have missed some auth/security/whatnot part of the wizard, because the new instance doesn't accept my SSH key.
[15:48:32] <kalle>	 Where should I look in Horizon? :)
[15:49:04] <andrewbogott>	 kalle: you shouldn't need to do anything special for a new VM — it does take 5+ minutes before all the ssh bits are set up though
[15:49:12] <andrewbogott>	 you can see progress in the instance log on Horizon
[15:49:19] <andrewbogott>	 (and also potential startup failures)
[15:49:46] <kalle>	 Aha I see, thanks!
[15:50:04] <andrewbogott>	 If the logs show things as all done and ready and you still can't connect, lmk and I can try to debug
[15:52:20] <kalle>	 andrewbogott: Thanks!
[16:00:38] <yuvipanda>	 andrewbogott: society is weird. millions of migrant workers are being forced to walk thousands of kilometers on almost no food or water to go back home
[16:00:55] <yuvipanda>	 andrewbogott: folks without privilege are just getting completely fucked over
[16:01:49] <andrewbogott>	 Hm, 'travel a long way to lots of different places' seems like the best way to spread a disease ever
[16:02:06] <andrewbogott>	 is there some actual rationale for that or is it just kneejerk xenophobia?
[16:03:07] <yuvipanda>	 andrewbogott: oh they have no money, and aren't being paid, so are walking to their home villages because they have no other way to survive
[16:04:27] <andrewbogott>	 well, that's awful
[16:18:52] <yuvipanda>	 yep
[17:39:59] <Cyberpower678>	 andrewbogott: where can I got to modify puppet roles?
[17:40:14] <Cyberpower678>	 horizon doesn't make it easy to find the location of the roles.
[17:41:05] <andrewbogott>	 Cyberpower678: are you finding the puppet tabs?
[17:41:16] <Cyberpower678>	 Yea, but it's uber simple
[17:42:20] <Cyberpower678>	 I want to see what's in the role, not what the instance has assigned to it.
[17:43:29] <bd808>	 Cyberpower678: https://github.com/wikimedia/puppet
[17:43:42] <andrewbogott>	 Ah — the structured puppet interface isn't available anymore (due to upstream project changes)
[17:43:47] <andrewbogott>	 you pretty much have to look at the source now
[17:44:54] <Cyberpower678>	 andrewbogott: bd808: and where would the roles be?
[17:45:20] <bd808>	 Cyberpower678: https://github.com/wikimedia/puppet/tree/production/modules/role/manifests
[17:45:45] <bd808>	 and also "profiles" at https://github.com/wikimedia/puppet/tree/production/modules/profile/manifests
[17:49:03] <Cyberpower678>	 bd808: andrewbogott: would any of you mind doing this change for me please. https://github.com/wikimedia/puppet/pull/6/files
[17:50:52] <kalle>	 I'm failing to apt install ca-certificates-java on a Debian buster instance due to what seems to be encoding problems on filesystem or in apt. I had the same problems on the old Debian 9.5 instance. 
[17:50:56] <kalle>	 https://pastebin.pl/view/8744020f
[17:51:29] <kalle>	 This is something I've only seen on wmflabs images. This is strange.
[17:53:01] <bd808>	 Cyberpower678: that needs to be a gerrit request as the bot told you. I don't have the superpowers to merge a puppet patch, but I can help find someone to do that once it is in gerrit
[17:53:17] * Cyberpower678 doesn't do gerrit
[17:53:30] <Cyberpower678>	 Gerrit confuses the hell out of me.
[17:53:34] <bd808>	 then you don't get the change?
[17:54:52] <Cyberpower678>	 bd808: I don't have any clue on how to use Gerrit.  The UI is already confusing enough as is.
[17:54:52] <bd808>	 kalle: that paste is missing any context of how you got that pem on disk
[17:55:44] <kalle>	 bd808: It's a brand new empty Debian buster image that I first added the /etc/environment on, and then apt install openjdk-11-jdk
[17:56:11] <kalle>	 ca-certificates-java is a dependency package of that.
[17:57:41] <Cyberpower678>	 Can ANYONE help me with pushing a simple change to puppet?
[17:59:19] <Cyberpower678>	 andrewbogott: ^
[18:01:13] <arturo>	 Cyberpower678: be patient please, people are working on other stuff
[18:01:22] <arturo>	 we don't usually work with github pullr equests
[18:02:30] <Cyberpower678>	 I can be patient, just as long as someone can help me with this in the end.
[18:02:53] <bd808>	 kalle: I'm looking at a random host, and I can see that same cert in /etc/ssl/certs but without any encoding weirdness. Can you tell me what instance you are working on and I can try to compare things to this other host.
[18:02:54] <Cyberpower678>	 IABot is down, and will not come back until I get the new puppet role pushed.
[18:03:09] <Cyberpower678>	 And if I can't push it, well, IABot stays off.
[18:04:44] <zhuyifei1999_>	 Cyberpower678: git commit -a; git review
[18:05:09] <arturo>	 Cyberpower678: I understand, but I'm sure you know people have different priorities
[18:05:43] <zhuyifei1999_>	 I think you should learn gerrit so you don't have to ping people for what you can do in a few clicks :P
[18:06:51] <zhuyifei1999_>	 the main difference between gerrit and github PRs, imo, is how gerrit likes to keep everything in one commit
[18:07:06] <zhuyifei1999_>	 so just --amend
[18:07:11] <Cyberpower678>	 zhuyifei1999_: Last time I tried to learn gerrit, I felt like I was trying to learn Japanese.  I wanted to fix an extension, and was so frustrated trying to push it, that I gave up along with developing on MW>
[18:07:28] <zhuyifei1999_>	 I can guide you
[18:07:36] <Cyberpower678>	 I would appreciate that.
[18:08:15] <zhuyifei1999_>	 ok so you got puppet cloned somewhere yet?
[18:08:37] <Cyberpower678>	 I can do that real quick.  Do I clone it from github, or...
[18:08:44] <zhuyifei1999_>	 gerrit, one sec
[18:09:28] <zhuyifei1999_>	 git clone "ssh://[your username]@gerrit.wikimedia.org:29418/operations/puppet"
[18:10:32] <Cyberpower678>	 Uhh, username
[18:10:37] <zhuyifei1999_>	 you can use github, but that adds an complexity when it creates a separate remote for gerrit, and then you'll have to fetch before `git review`
[18:10:48] <zhuyifei1999_>	 yeah, your username?
[18:11:09] <Cyberpower678>	 Do I need a gerrit account, or is it LDAPed with something?
[18:11:16] <zhuyifei1999_>	 it's LDAP-ed
[18:11:23] <zhuyifei1999_>	 so the LDAP username
[18:11:40] <zhuyifei1999_>	 like, when you ssh to iabot, the username
[18:12:49] <Cyberpower678>	 Got it
[18:13:02] <Cyberpower678>	 Alright let me try to clone it.
[18:13:05] <zhuyifei1999_>	 k
[18:13:54] <Cyberpower678>	 Permission denied.
[18:14:01] <zhuyifei1999_>	 key issue?
[18:14:04] <Cyberpower678>	 I assume it's the same key as on my VPS?
[18:14:23] <zhuyifei1999_>	 you need to upload it to https://gerrit.wikimedia.org/r/#/settings/ssh-keys
[18:14:29] <zhuyifei1999_>	 no
[18:14:37] <Cyberpower678>	 Okay, that's why
[18:14:47] <zhuyifei1999_>	 for some reason it doesn't look up key from ldap
[18:16:27] <Cyberpower678>	 That's unfortunate.
[18:16:52] * Cyberpower678 has a convoluted key setup here, so hold on.
[18:17:01] <zhuyifei1999_>	 ok lol
[18:18:00] <Cyberpower678>	 I have a bunch of fake id_rsa files
[18:21:02] * zhuyifei1999_ has too many real keys, to the point that my ssh config needs `IdentitiesOnly yes` or else a ssh server would think I'm bruteforcing keys and reject my login
[18:22:32] <zhuyifei1999_>	 (probably better now that I don't use gnome anymore and keys don't auto add to ssh agent)
[18:24:00] <zhuyifei1999_>	 Cyberpower678: if you have trouble getting your public key, I can do a lookup from ldap
[18:24:15] <Cyberpower678>	 Sure.
[18:25:34] <Cyberpower678>	 zhuyifei1999_: alright, I'm in
[18:25:41] <Cyberpower678>	 It's cloning
[18:25:43] <zhuyifei1999_>	 ok
[18:26:23] <Cyberpower678>	 Hey cool, GitHub desktop took it in.
[18:27:02] <zhuyifei1999_>	 uh, no idea what that means
[18:28:21] <zhuyifei1999_>	 oh you are on windows right? you need https://www.mediawiki.org/wiki/Gerrit/git-review#Windows
[18:28:28] <Cyberpower678>	 It popped up on GitHub desktop which is what I prefer to use, as I'm a visual guy.
[18:28:36] <Cyberpower678>	 No macOS
[18:28:52] <zhuyifei1999_>	 ok
[18:29:30] <zhuyifei1999_>	 https://www.mediawiki.org/wiki/Gerrit/git-review#Method_1_(Homebrew)
[18:29:45] <zhuyifei1999_>	 so do the brew install git-review
[18:33:46] <Cyberpower678>	 zhuyifei1999_: I made the modifications to the necessary file, but I can't seem to push it.
[18:34:00] <zhuyifei1999_>	 run `git review` instead of `git push`
[18:36:27] <Cyberpower678>	 git review is still installing
[18:37:04] <zhuyifei1999_>	 ok
[18:40:08] <Cyberpower678>	 Ugh.  I got permission denied ago.  It's rejecting my key.
[18:40:23] <Cyberpower678>	 But I loaded in my public key.
[18:40:31] <Cyberpower678>	 And it was able to successfully clone.
[18:41:00] <zhuyifei1999_>	 can I see where the clone output and whatever is emitting permission denied?
[18:41:42] <Cyberpower678>	 I ran it again and it logged in.  
[18:41:46] <Cyberpower678>	  ! [remote rejected]       HEAD -> refs/for/production (commit 1623684: invalid committer)
[18:41:47] <Cyberpower678>	 error: failed to push some refs to 'ssh://cyberpower678@gerrit.wikimedia.org:29418/operations/puppet.git'
[18:42:07] <Cyberpower678>	 But I can't git review it either.
[18:42:39] <zhuyifei1999_>	 what's the output of `git review`?
[18:42:51] <zhuyifei1999_>	 oh I might know why
[18:43:20] <zhuyifei1999_>	 try add your email to https://gerrit.wikimedia.org/r/#/settings/contact
[18:43:34] <zhuyifei1999_>	 it wants to make sure you are committing with your email
[18:44:07] <Cyberpower678>	 zhuyifei1999_: it's already in there.
[18:44:37] <zhuyifei1999_>	 and the output is 'invalid committer'?
[18:44:45] <Cyberpower678>	 yes
[18:45:06] <zhuyifei1999_>	 does `git log` show the right email for the author?
[18:45:35] <Cyberpower678>	 Oh.  I think I know what's wrong.  My primary email is not the same on git as it is on gerrit.
[18:46:15] <zhuyifei1999_>	 ok, so set your email in git
[18:46:49] <zhuyifei1999_>	 then `git commit --amend --reset-author`
[18:46:56] <Cyberpower678>	 My email is set to what's on GitHub, I really don't want to change that.
[18:47:17] <zhuyifei1999_>	 you can do per-repo config
[18:47:41] <Cyberpower678>	 I think it's jsut easier to add an email to Gerrit, but it won't let me.
[18:48:05] <zhuyifei1999_>	 use --local instead of --global for per-repo config
[18:48:07] <Cyberpower678>	 I click register new email, type it in, click register, and then I'm logged out.
[18:48:19] <zhuyifei1999_>	 huh that sounds like a bug
[18:50:15] <Cyberpower678>	 Happens on Chrome too.  I can't change my email on Gerrit. :/
[18:50:35] <zhuyifei1999_>	 file a bug?
[18:51:02] <zhuyifei1999_>	 for now you can do the `git config --local user.email you@example.com`
[18:51:07] <Cyberpower678>	 FINALLY
[18:51:09] <Cyberpower678>	 It registered
[18:51:12] <zhuyifei1999_>	 ok
[18:51:54] <Cyberpower678>	 Only had to do it like 6 times. :p
[18:52:10] <zhuyifei1999_>	 does `git review` work now?
[18:52:38] <Cyberpower678>	 Looks like it?
[18:53:56] <zhuyifei1999_>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/596497/
[18:53:59] <Cyberpower678>	 zhuyifei1999_: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/596497
[18:54:00] <Cyberpower678>	 YAY
[18:54:02] <Cyberpower678>	 :D
[18:54:08] <zhuyifei1999_>	 congrats
[18:54:22] <zhuyifei1999_>	 now /me gotta get food
[18:54:29] <Cyberpower678>	 THANKS
[18:54:33] <zhuyifei1999_>	 np
[18:54:54] <Cyberpower678>	 bd808: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/596497 okay it's up now. :p
[19:10:07] <Cyberpower678>	 bstorm_: when will this change go live?
[19:10:53] <bstorm_>	 It's merging...then it'll be a puppet run. Depends on where that is applied
[19:11:23] <bstorm_>	 But it's merged and live now on production, there's sometimes a delay for that to sync over to all cloud hosts
[19:11:30] <bstorm_>	 on the order of tens of minutes
[19:23:19] <Cyberpower678>	 bstorm_: okay.  I'm going to go ahead and backup my exec VM and scrap it.
[19:23:50] <Cyberpower678>	 bstorm_: before I do, does the change to the puppet role automatically change the environment that it's applied to?
[19:25:11] <bstorm_>	 I'm not 100% sure I understand your question, but when the puppet role is changed it applies to puppet on the server where the role is applied. That means that the role should affect any server with the role applied (existing or new) as long as the puppetmaster has the change.
[19:25:50] <bstorm_>	 puppet tends to run every 20 min or so if you don't run it by hand
[19:26:04] <bstorm_>	 And it also runs on build of a new VM
[19:26:56] <Cyberpower678>	 bstorm_: I changed the PHP version to 7.2.  Will puppet change PHP on my VM is what I was asking?
[19:30:02] <bstorm_>	 I don't know your puppetization that well. It looks like it will either do that or it will not work without an additional apt repo from what I can tell
[19:30:15] <bstorm_>	 It will almost certainly leave php 7.0 behind, though either way
[19:30:32] <bstorm_>	 So you may need to manually remove that after confirming that 7.2 is installed
[19:30:39] <Cyberpower678>	 Well.  I can't get apt to pick up PHP 7.2 at all even with the added repo.
[19:30:43] <bstorm_>	 unless it is a new VM...then it should just install 7.2
[19:32:32] <bstorm_>	 Cyberpower678: what do you get from `sudo apt-cache policy php7.2-cli`
[19:38:23] <Cyberpower678>	 N: Unable to locate package php7.2-cli
[19:38:23] <Cyberpower678>	 N: Couldn't find any package by glob 'php7.2-cli'
[19:38:23] <Cyberpower678>	 N: Couldn't find any package by regex 'php7.2-cli'
[19:38:26] <Cyberpower678>	 bstorm_: ^
[19:38:44] <bstorm_>	 They may not exist. Some packages changed names a bit more
[19:39:13] <bstorm_>	 No that exists
[19:39:26] <bstorm_>	 That suggests you don't have the repo on the server or haven't run apt-get update since adding it
[19:39:45] <bstorm_>	 we get it in toolforge from:
[19:39:57] <bstorm_>	 https://www.irccloud.com/pastebin/n3VqgC0a/
[19:40:30] <Cyberpower678>	 bstorm_: I have, and it still won't find it.  Which repo are you using?
[19:41:23] <bstorm_>	 https://www.irccloud.com/pastebin/Ju5LhKiP/
[19:41:33] <bstorm_>	 That's what that puppet creates
[19:42:57] <Cyberpower678>	 My VM is still chilling at 7.0
[19:48:37] <bstorm_>	 Do you have the file /etc/apt/sources.list.d/php72-external-stretch.list?
[19:49:33] <bstorm_>	 more to the point, if you run `grep php72 /etc/apt/sources.list.d/*` do you get anything?
[19:50:10] <Cyberpower678>	 Hmm, nope.  As a matter of fact, that directory is almost empty.
[19:55:08] <bstorm_>	 Then you will need to add the repo. The puppet version is what I pasted above 
[19:55:23] <bstorm_>	 Eating and maybe I can help with that in a bit
[19:56:31] <Cyberpower678>	 bstorm_: I'm actually inclined to respawn the VM, since the image is out of date anyway.
[19:56:43] <Cyberpower678>	 But I still welcome help.
[20:27:31] <Cyberpower678>	 bstorm_: am I missing a step.  I respawned my node with a new image, but I can't SSH into it.
[20:27:40] <Cyberpower678>	 I keep getting access denied
[20:28:23] <Cyberpower678>	 nvm
[20:28:34] <bd808>	 Cyberpower678: have you checked the "logs" tab on Horizon to see if Puppet has finished its initial run yet? It can take 10-15 minutes to finish
[20:29:11] <Cyberpower678>	 Yea, it finished, but it didn't install PHP onto the VM
[20:29:23] <Cyberpower678>	 Despite being instructed to.
[20:30:16] <bd808>	 `sudo -i puppet agent -tv` and look for error messages
[20:32:15] <Cyberpower678>	 Yea, a lot of error
[20:33:02] <Cyberpower678>	 And all of them being unable to locate php7.2 and its modules.
[20:34:31] <Cyberpower678>	 It does have PHP 7.3 in the repo
[20:34:38] <Cyberpower678>	 I guess I'll use that.
[20:35:10] <bd808>	 Cyberpower678: did you build a Buster instance? If so then yes PHP 7.3 should be the PHP to use
[20:36:30] <Cyberpower678>	 As long as it's higher than php 7.1
[20:36:46] <Cyberpower678>	 -)
[20:36:48] <Cyberpower678>	 :-)
[20:37:12] * Cyberpower678 needs to reinstall all of the files he zipped up and saved
[20:38:25] <Cyberpower678>	 Yikes, transfer speeds are abysmal.
[20:38:29] <Cyberpower678>	 2 MB/s
[21:12:52] <andrewbogott>	 !log admin rebuilding cloudvirt1003-wdqs as part of T252831
[21:12:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[21:12:56] <stashbot>	 T252831: cloudvirt ceph nodes can't launch new VMs - https://phabricator.wikimedia.org/T252831
[22:15:10] <bstorm_>	 !log admin changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 T252831 
[22:15:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[22:15:13] <stashbot>	 T252831: cloudvirt ceph nodes can't launch new VMs - https://phabricator.wikimedia.org/T252831
[22:21:04] <bstorm_>	 !log admin upgrading qemu-system-x86 on cloudvirt1006 to backports version T252831
[22:21:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[22:21:08] <stashbot>	 T252831: cloudvirt ceph nodes can't launch new VMs - https://phabricator.wikimedia.org/T252831
[22:26:46] <Jayprakash12345>	 !log tools.indic-wscontest [Bug] Use validation timestamp instead of proofread timestamp (T252834)
[22:26:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.indic-wscontest/SAL
[22:26:48] <stashbot>	 T252834: indic-wscontest: Validation count is not correct - https://phabricator.wikimedia.org/T252834
[23:10:40] <Cyberpower678>	 andrewbogott: is the new cyberbot-exec-iabot-01 sitting on a virtcluster that is constrained?
[23:11:16] <Cyberpower678>	 What took 50% CPU before, is now using 100% and the network IO appears to be limited.
[23:25:18] <bstorm_>	 Looks like it's on cloudvirt1016 https://openstack-browser.toolforge.org/server/cyberbot-exec-iabot-01.cyberbot.eqiad.wmflabs. That host doesn't look particularly constrained https://grafana.wikimedia.org/d/000000576/cloud-capacity-planning?orgId=1&var-cloudvirt=cloudvirt1016:9100
[23:26:07] <bstorm_>	 Load is much higher on some other virts.
[23:28:53] <bstorm_>	 !log admin downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time T252831
[23:28:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[23:28:56] <stashbot>	 T252831: cloudvirt ceph nodes can't launch new VMs - https://phabricator.wikimedia.org/T252831