[10:06:00] <arturo>	 !log mailman created & delegated DNS zone `lists.wmcloud.org` (T278358)
[10:06:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mailman/SAL
[10:06:04] <stashbot>	 T278358: Delegate lists.wmcloud.org domain to be able to add DNS DKIM records - https://phabricator.wikimedia.org/T278358
[10:08:49] <dcaro>	 !log admin upgrading kernel on cloudcephmon2003-dev and reboot (T274565)
[10:08:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:08:53] <stashbot>	 T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565
[10:18:23] <dcaro>	 !log admin upgrading kernel on cloudcephosd2002-dev and reboot (T274565)
[10:18:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:18:28] <stashbot>	 T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565
[10:24:10] <dcaro>	 !log admin upgrading kernel on cloudcephosd2003-dev and reboot (T274565)
[10:24:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:24:14] <stashbot>	 T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565
[10:31:08] <dcaro>	 !log admin kernel upgrade on osds on codfw done, running performance tests (T274565)
[10:31:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:31:13] <stashbot>	 T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565
[12:51:05] <arturo>	 !log tools create cinder volume `tools-aptly-data` (T278354)
[12:51:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:51:11] <stashbot>	 T278354: Toolforge: migrate services node to Debian Buster - https://phabricator.wikimedia.org/T278354
[12:58:38] <arturo>	 !log tools created VM `tools-services-05` as Debian Buster (T278354)
[12:58:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:58:43] <stashbot>	 T278354: Toolforge: migrate services node to Debian Buster - https://phabricator.wikimedia.org/T278354
[13:31:08] <arturo>	 !log tools point aptly clients to `tools-services-05.tools.eqiad1.wikimedia.cloud` (hiera change) (T278354)
[13:31:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:31:14] <stashbot>	 T278354: Toolforge: migrate services node to Debian Buster - https://phabricator.wikimedia.org/T278354
[13:33:39] <arturo>	 !log tools shutdown tools-sge-services-04 (T278354)
[13:33:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:36:37] <arturo>	 !log tools shutdown tools-sge-services-03 (T278354)
[13:36:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:36:42] <stashbot>	 T278354: Toolforge: migrate services node to Debian Buster - https://phabricator.wikimedia.org/T278354
[14:31:04] <thesocialdev>	 Hi everyone, here I am again. But now with trouble to set a puppet master following the tutorial https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster
[14:32:25] <thesocialdev>	 when trying to do this for the instance maps-puppetmaster01.maps-experiments.eqiad1.wikimedia.cloud apache2 wasn't starting
[14:35:35] <thesocialdev>	 are there any reason for that? I'm going to destroy and start a new instance in a few minutes and retry the tutorial. But I thought you would like to know what the error was when running the puppet agent twice in the fresh puppetmaster machine
[14:36:30] <thesocialdev>	 https://www.irccloud.com/pastebin/eoSMBn4R/
[14:38:15] <Majavah>	 thesocialdev: what's in the apache2 syslog? that's just saying "it failed" but the syslog likely contains why
[14:38:42] <thesocialdev>	 https://www.irccloud.com/pastebin/yeSmUaiO/
[14:39:43] <Majavah>	 uhh, what about /var/log/apache2/error.log?
[14:41:40] * Majavah needs to leave for a moment, hopefully others are around to help
[14:41:51] <thesocialdev>	 sorry, I'm restarting the process and I have deleted the instance. If it happens again I'll post more information.
[15:16:25] <bstorm>	 !log tools.refill-api deleting pods to restart service T278211
[15:16:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.refill-api/SAL
[15:16:30] <stashbot>	 T278211: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T278211
[15:20:53] <bstorm>	 thesocialdev: The puppetization for standalone may have been affected by upstream changes. The error you were seeing sounds like https://stackoverflow.com/questions/48525092/docker-httpd-configuration-error-no-mpm-loaded The fact that there's an error at all suggests that maybe the puppet role needs updates. All else fails, you might be able to update the config by hand to fix it at first
[15:23:14] <thesocialdev>	 thanks bstorm I wasn't sure about what was going on and tried to recreate a new puppetmaster, right now I'm facing an odd behaviour for the new instance, it's scheduled to build for the past 40 minutes. Once it's up I'll try again the process and if the problem happen again I'll fix it locally.
[15:24:06] <bstorm>	 If it isn't building, that sounds like there's some error or something...are you over quota?
[15:27:54] <thesocialdev>	 bstorm: quota is fine. I assumed because I created and deleted an instance quickly I was facing some rate limiting
[15:28:16] <bstorm>	 That shouldn't trigger rate limiting
[15:28:32] <bstorm>	 which project?
[15:28:45] <bstorm>	 oh I see maps-experiments
[15:30:21] <bstorm>	 thesocialdev: there's something broken there that we'll fix. I'm running to a meeting atm
[15:30:42] <thesocialdev>	 oh okay, thanks bstorm 
[15:31:32] <andrewbogott>	 thesocialdev: I'll take a look, I've been working on new builds over the last few days
[15:33:41] <thesocialdev>	 thanks @andrewbogott 
[15:35:14] <andrewbogott>	 thesocialdev: I think you should just kill it and try again.  I'm curious but you don't need to wait for me to satisfy my curiosity unless it happens repeatedly.
[15:36:42] * thesocialdev recreating instance
[15:37:05] <thesocialdev>	 yeah @andrewbogott no issue now, instance created
[15:37:13] <andrewbogott>	 :/
[15:37:58] <andrewbogott>	 thesocialdev: I think maybe you had a name collision, did you delete a VM with the same name before creating the stuck one?
[15:38:49] <thesocialdev>	 can the name collision happen even if the build doesn't happen?
[15:39:21] <thesocialdev>	 I have changed the name when trying to create the instance that failed to build
[15:39:24] <andrewbogott>	 probably — but I'm talking about the stuck build from an hour ago not the one you did just now (which seems fine)
[15:39:51] <andrewbogott>	 There are some race conditions; it's best to wait a few minutes after deleting a VM before recreating with the same name.
[15:39:57] <andrewbogott>	 I'm not 100% sure that's what happened
[15:40:04] <andrewbogott>	 But anyway, it sounds like you're un-stuck now
[15:40:16] <thesocialdev>	 That would be weird because I changed the name, I don't think the collision could happen in the first time
[15:40:32] <thesocialdev>	 Yes, unblocked until proved wrong. Thanks!
[16:05:26] <bstorm>	 !log tools failed over the tools grid to the shadow master T277653
[16:05:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:05:32] <stashbot>	 T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653
[16:18:18] <arturo>	 !log tools icinga-downtime toolschecker for 2h
[16:18:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:20:20] <arturo>	 !log tools rebuilding tools-sgegrid-master VM as debian buster (T277653)
[16:20:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:20:24] <stashbot>	 T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653
[16:30:05] <bd808>	 !log tools.openstack-browser Added Majavah as co-maintainer
[16:30:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL
[16:38:10] <wm-bot>	 I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 2.8.1.0 [libirc v. 1.0.3] my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features
[16:38:10] <arturo>	 @help
[16:38:13] <arturo>	 @op
[16:55:06] <thesocialdev>	 and here I am again on for maps-experiments. I can't get the certificates to validate following https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client
[16:55:30] <thesocialdev>	 after doing every step without errors, I run puppet agent one last time and get `certificate verify failed (certificate revoked)`
[16:58:49] <Majavah>	 thesocialdev: are you re-using an instance name? is that on the global puppetmaster or a project-local one?
[16:59:24] <thesocialdev>	 I created maps-puppetmaster02.maps-experiments.eqiad1.wikimedia.cloud
[16:59:29] <Majavah>	 !log tools.openstack-browser-dev deploying https://phabricator.wikimedia.org/D1190
[16:59:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser-dev/SAL
[16:59:52] <thesocialdev>	 and I'm trying to point the other instances in the maps-experiments project to the new puppet-master 
[17:00:48] <thesocialdev>	 an useful information could be: in the puppet-master apache2 was failing because the conf file is being truncated by upstream
[17:01:10] <thesocialdev>	 I purged and reinstalled apache2 and it worked again
[17:02:16] <Majavah>	 are they pointing to the correct puppetmaster? `grep master /etc/puppet/puppet.conf`
[17:02:32] <thesocialdev>	 Majavah yes
[17:03:16] <thesocialdev>	 https://www.irccloud.com/pastebin/uqh2IPKM/
[17:03:56] <thesocialdev>	 I can run every command on step 2 https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client
[17:04:25] <thesocialdev>	 when I run puppet agent for the last time in the client, it errors with certificate revoked
[17:05:44] <Majavah>	 weird, on the puppetmaster I'd try `sudo puppet cert clean full-instance-name`
[17:06:23] <thesocialdev>	 I did that, and repeat the procedure
[17:06:25] <thesocialdev>	 same result
[17:08:16] <Majavah>	 hmm, in that case I have no ideas what's causing this :(
[17:08:48] <Majavah>	 !log tools.openstack-browser deploying https://phabricator.wikimedia.org/D1190
[17:08:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL
[17:15:43] <andrewbogott>	 !log admin refreshing puppet compiler facts for tools project
[17:15:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[17:46:10] <arturo>	 !log tools rebooting tools-sgeexec-* nodes to account for new grid master (T277653)
[17:46:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:46:17] <stashbot>	 T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653
[18:20:48] <musikanimal>	 job ID 9999796 on tools.svgtranslate appears to be stuck in the deletion state. Can a Toolforge admin kill it pweez?
[18:26:50] <Majavah>	 bstorm: repeating musikanimal request now that you see this channel: "job ID 9999796 on tools.svgtranslate appears to be stuck in the deletion state. Can a Toolforge admin kill it pweez?"
[18:27:22] <bstorm>	 Yup, I was split from the channel before, so I didn't see it
[18:27:36] <bstorm>	 I'll get it...and probably others if I can find em
[18:27:55] <bstorm>	 done
[18:27:58] <bstorm>	 on that one
[18:34:51] <musikanimal>	 job 9999796 on tools.svgtranslate still appears stuck on my end. Looks like this might be a widespread issue? I have 4 jobs stuck on tools.musikbot as well
[18:45:29] <andrewbogott>	 musikanimal: there was a grid outage; you may need to stop/restart things
[18:46:34] <musikanimal>	 I see. I tried running `qdel` on these jobs, they are stuck in the `dr` state. When this happened before I needed someone with root to intervene
[18:46:55] <andrewbogott>	 ok — that might be a bstorm question
[18:47:00] <bstorm>	 musikanimal: I'll take a look in a couple min
[18:47:30] <musikanimal>	 okay thanks! for tools.musikbot it's jobs 9999690, 9999770, 9999771 and 9999866
[19:03:40] <andrewbogott>	 !log admin deleting all unused (per wmcs-imageusage) Jessie base images from Glance 
[19:03:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[19:25:02] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 77328e559d (optional forms)
[19:25:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[19:26:54] <bstorm>	 musikanimal: forcing deletion is working, so I'm killing the jobs now
[19:30:02] <musikanimal>	 thanks!
[19:30:39] <bstorm>	 !log tools forced deletion of all jobs stuck in a deleting state T277653
[19:30:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:30:44] <stashbot>	 T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653
[19:30:51] <bstorm>	 I found a bunch of other ones
[19:34:39] <bstorm>	 It looks like all jobs are now running or forced to wait because the user launched more than their limit of jobs (which is fine).
[19:36:40] <gifti>	 i think 1492646 and 3608591 are stuck as well
[19:46:48] <bstorm>	 Yeah, I see the d status...why is that showing up now and not before?
[19:46:55] <bstorm>	 I had grepped for it hrm
[19:46:57] <gifti>	 i deleted them
[19:47:00] <gifti>	 after that
[19:47:29] <bstorm>	 That makes me wonder if deletion isn't working for some reason
[19:47:44] <gifti>	 i deleted three other jobs, that worked fine
[19:47:52] <bstorm>	 That's good :)
[19:48:49] <bstorm>	 Were they also on continuous queues?
[19:49:01] <gifti>	 yes, all of them
[19:49:12] <bstorm>	 Ok, well, I force deleted them
[19:49:31] <gifti>	 great, thanks
[19:49:39] <bstorm>	 Hopefully there isn't some odd pattern lingering or an exec node that is still suffering some kind of issue.
[19:49:45] <bstorm>	 I'll keep an eye out
[19:50:20] <bstorm>	 looks healthy again atm
[20:25:54] <bd808>	 !log tools.wikibugs Force deleted stuck wb2-grrrrit job 22577
[20:25:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[20:26:15] <Majavah>	 !logs tools.wikibugs restart gerrit listener
[20:27:02] <bd808>	 Majavah: "!logs" typo there ^
[20:27:18] <Majavah>	 bd808: oops, thanks
[20:27:43] <Majavah>	 !log tools.wikibugs restart gerrit listener
[20:27:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[20:57:17] <bd808>	 !log toolhub Updated to bd808/toohub-beta:search @ 0c778de74c7a
[20:57:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolhub/SAL
[22:01:01] <bstorm>	 !log quarry restarting web interface for a small fix for the database field display T264254
[22:01:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[22:01:06] <stashbot>	 T264254: Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254
[22:02:59] <bstorm>	 !log quarry restarting celery worker processes to fix connection cleanup T264254
[22:03:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[22:15:30] <bstorm>	 !log quarry removing the querykiller role T264254
[22:15:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[22:15:34] <stashbot>	 T264254: Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254
[22:23:04] <legoktm>	 wow Python 3 is up to 67%
[22:24:07] <legoktm>	 I'm not sure I believe JavaScript is only 3% though
[22:29:34] <legoktm>	 oh, it was 68% last year, I missed that
[23:18:29] <bstorm>	 :)