[04:38:57] <vgutierrez>	 _joe_: I'm not seeing blubberoid listed on T210411
[04:39:19] <vgutierrez>	 does it mean that a TLS endpoint is available for blubberoid?
[04:41:51] <vgutierrez>	 no cert on the puppet repo.. I assume that somebody forgot about poor old blubberoid on T210411
[05:01:32] <_joe_>	 vgutierrez: yeah
[05:01:39] <_joe_>	 I think that's the case
[05:01:56] <vgutierrez>	 even the mapping rules were forgotten so...
[05:02:03] <_joe_>	 I mean we can connect without TLS for now, and I can add a TLS termination image
[05:02:07] <vgutierrez>	 anyways.. the issue has been solved anyways
[05:02:11] <_joe_>	 to the chart
[05:02:14] <vgutierrez>	 yup
[09:07:14] <wikibugs>	 10serviceops, 10Operations, 10Release Pipeline, 10local-charts, and 3 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Joe)
[10:46:24] <wikibugs>	 10serviceops, 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) Almost there:   > ` >  >  > Config error: Cannot open config file "/etc/bacula/bacula-sd.conf": Permission denied >  > `
[19:56:28] <hashar>	 mutante: thx for the zuul mask :D
[19:56:34] <hashar>	 I should have done the puppet patch as soon as you suggested the solution
[19:59:20] <mutante>	 hashar: you're welcome. i forgot but something about it was confusing. maybe what you fixed in commit message. anyways it works as expected on 2001 :)
[19:59:29] <mutante>	 and nothing happened on 1001
[19:59:50] <hashar>	 awesome
[19:59:59] <hashar>	 how can you tell the service is masked?
[20:01:07] <mutante>	 well. i only saw puppet run the command during run
[20:01:11] <mutante>	 but.. ehm   [contint2001:~] $ sudo systemctl list-units --all --state=masked
[20:01:15] <mutante>	 wait a sec :)
[20:01:19] <hashar>	 $ systemctl status zuul
[20:01:20] <hashar>	 ● zuul.service - Zuul Server
[20:01:20] <hashar>	    Loaded: loaded (/lib/systemd/system/zuul.service; enabled)
[20:01:20] <hashar>	    Active: inactive (dead) since Fri 2019-09-20 08:27:17 UTC; 11h ago
[20:01:34] <hashar>	 it is not like I understand anything about systemd services status ;]
[20:02:02] <hashar>	 or maybe the daemon has to be reloaded?
[20:03:13] <mutante>	 so it was already 'dead' since 11hours ago
[20:03:58] <mutante>	 reloading it fails with reason "it is inactive"
[20:04:42] <mutante>	 which is kind of what we want.. but what i really want is to see it in the list of masked units
[20:05:47] <mutante>	 hashar: if i manually run the mask command i get it to show up
[20:05:53] <mutante>	 [contint2001:~] $ sudo systemctl list-units --all --state=masked
[20:06:16] <hashar>	 ah now it shows
[20:06:19] <mutante>	 running the command meant "Created symlink from /etc/systemd/system/zuul.service to /dev/null.
[20:07:02] <mutante>	 but that didn't happen automatically ..even though i saw puppet run the exec
[20:07:05] <hashar>	 but you had to run the command manually?
[20:07:07] <hashar>	 :-\
[20:07:08] <mutante>	 yea..
[20:07:08] <hashar>	 fun
[20:07:39] <hashar>	 maybe puppet --debug would give more details
[20:08:58] <mutante>	 command => "/bin/systemctl mask ${unit}",
[20:09:05] <mutante>	 that's what i did ...  trying to debug
[20:11:07] <mutante>	 Sep 20 20:08:41 contint2001 puppet-agent[30969]: Executing: '/bin/systemctl unmask zuul'
[20:16:08] <mutante>	 Notice: /Stage[main]/Zuul::Server/Systemd::Service[zuul]/Service[zuul]/enable: enable changed 'false' to 'true'
[20:17:29] <mutante>	 unmask gets executed "before" zuul::server class.. and inside zuul::server $service_enable defaults to true ?
[20:17:36] <mutante>	 so puppet enables it again
[20:17:43] <hashar>	 :((
[20:17:51] <hashar>	 maybe because of the unless =>  statement ? 
[20:17:52] <mutante>	 sorry, mask gets executed ...
[20:18:01] <hashar>	 :-\
[20:18:24] <hashar>	 so even with a test against the catalogue I got it wrong somehow :-\
[20:18:44] <mutante>	 yea, i can confirm it looked right in compiler
[20:24:48] <mutante>	 we could move the entire ::zuul::server inside the  else branch
[20:25:00] <hashar>	 that sounds odd
[20:25:18] <hashar>	 oh
[20:25:19] <hashar>	 yeah
[20:25:19] <mutante>	 but if it's disabled then we just want it to mask and do nothing else
[20:25:20] <hashar>	 so 
[20:25:32] <hashar>	 systemd::mask zuul.service
[20:25:40] <hashar>	 and AFTER that we get Class::Zuul::Server
[20:25:47] <hashar>	 which must define a Service[] 
[20:25:51] <hashar>	 which then reenable it
[20:25:52] <hashar>	 racy
[20:26:19] <mutante>	 yes, it works for a split second and then it gets enabled again
[20:26:24] <hashar>	 so my patch is wrong
[20:26:28] <hashar>	 we should just revert it
[20:26:32] <mutante>	 definitely could confirm that by stopping puppet agent in the right moment
[20:27:37] <hashar>	 modules/zuul/manifests/server.pp is borked
[20:27:43] <hashar>	     systemd::service { 'zuul':
[20:27:43] <hashar>	         ensure         => 'present',
[20:27:57] <hashar>	         service_params => {
[20:27:57] <hashar>	             ensure     => ensure_service($service_ensure),
[20:28:01] <hashar>	 so that is two ensure!?!
[20:28:43] <hashar>	 and that ensure_service only allows  running or stopped
[20:29:24] <mutante>	 yea
[20:29:40] <mutante>	 i would say if zuul is set to be disabled.. we dont need to use this class at all
[20:29:49] <mutante>	 all we need to do is run the mask command
[20:31:18] <hashar>	 meanwhile https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/538329/ reverts my faulty change :D
[20:32:24] <mutante>	 hashar: https://gerrit.wikimedia.org/r/c/operations/puppet/+/538330/2/modules/profile/manifests/zuul/server.pp  ?   
[20:32:35] <mutante>	 if enable = false .. then mask it and nothing else
[20:32:43] <mutante>	 otherwise.. use the class as before
[20:33:29] <hashar>	 fun,  modules/zuul/manifests/server.pp
[20:33:43] <hashar>	 it has a service_enable  which is no more used
[20:34:59] <mutante>	 well, that also fails because " service disabled should contain Systemd::Mask[zuul.service] that comes before Class[Zuul::Server]"
[20:35:31] <hashar>	 yeah
[20:35:32] <hashar>	 then 
[20:35:42] <hashar>	 your change means the zuul::server is no more applied
[20:36:30] <mutante>	 yea, but we don't really want to apply it if all we care is that it does NOT run
[20:36:41] <mutante>	 that symlink should already do it
[20:37:04] <hashar>	 and of course systemd::service does not support masked :\
[20:37:23] <mutante>	 yea. we got here before i think
[20:39:18] <mutante>	 i noticed that $service_enable as well, easy to mix it up with $service_ensure which is still used
[20:40:23] <hashar>	 yeah those are too complicated
[20:40:35] <hashar>	 but 
[20:40:54] <hashar>	 maybe one can do service { 'zuul': ensure => masked }
[20:41:46] <mutante>	 service enable => mask 
[20:41:58] <hashar>	 ah
[20:42:07] <mutante>	 allowed values for "enable" are true, false, mask or manual
[20:43:13] <hashar>	 so we can totally revert my patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/538329/
[20:43:47] <mutante>	 ok, doing
[20:47:16] <mutante>	 so let's try if $service_enable == false {  ... service { 'zuul': enable = 'mask' }  ?  i need to go on a break first though
[20:47:30] <mutante>	 revert is applied
[20:47:51] <hashar>	 cool
[20:47:58] <mutante>	 i also ran the mask command manually again
[20:48:23] <hashar>	 and probably systemd::mask and systemd::unmask  should never be used
[20:48:31] <hashar>	 since the Service[] would effectively override them
[20:48:33] <mutante>	 Notice: /Stage[main]/Zuul::Server/Systemd::Service[zuul]/Service[zuul]/enable: enable changed 'false' to 'true'
[20:48:38] <hashar>	 ;D
[20:48:45] <hashar>	 false?
[20:49:04] <hashar>	 and eventually I went with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/538331/
[20:49:30] <hashar>	 but that might be wrong
[20:49:53] <hashar>	 so that profile::zuul::server::service_enable is a boolean
[20:50:01] <hashar>	 and setting it to false, would cause the service to be masked
[20:50:02] <hashar>	 pff
[20:50:05] <hashar>	 yeah puppet!
[20:50:10] <mutante>	 it needs to be just "mask" instead of "masked" afaict
[20:50:19] <mutante>	 https://puppet.com/docs/puppet/5.5/types/service.html#service-attribute-enable
[20:51:31] <hashar>	 amended
[20:51:45] <mutante>	 default => true ?
[20:51:48] <hashar>	 and  enable => present did not exist
[20:51:51] <hashar>	 based on that doc
[20:51:53] <hashar>	 it is "true"
[20:52:01] <mutante>	 yea, that
[20:52:25] <hashar>	 but that patch would not have any spec test added hehe
[20:54:43] <mutante>	 seems to work  https://puppet-compiler.wmflabs.org/compiler1001/18471/
[20:55:44] <hashar>	 yeah
[20:55:47] <mutante>	 i'll just try it 
[20:56:05] <hashar>	 and certainly systemd::mask and systemd::unmask should be removed
[20:56:30] <hashar>	 since puppet does support that already via  service { enable => 'mask' }
[20:56:34] <hashar>	 and ervice { enable => true }
[20:57:28] <mutante>	 contint2001:  Notice: /Stage[main]/Zuul::Server/Systemd::Service[zuul]/Service[zuul]/enable: enable changed 'true' to 'mask'
[20:57:47] <mutante>	 but does not mean it shows up in the list .. 
[20:59:28] <mutante>	 but we do have the symlink: 
[20:59:28] <mutante>	 zuul.service: symbolic link to /dev/null
[21:00:16] <mutante>	 looks fine:
[21:00:17] <mutante>	 sudo systemctl status zuul | grep masked
[21:00:17] <mutante>	    Loaded: masked (/dev/null)
[21:00:28] <mutante>	 still does not show it:
[21:00:29] <mutante>	 sudo systemctl list-units --all --state=masked
[21:00:37] <hashar>	 but it is masked isnt' it?
[21:00:43] <mutante>	 status says yes
[21:00:59] <mutante>	 --state=masked says no ..
[21:01:12] <hashar>	 Sep 20 20:03:32 contint2001 systemd[1]: Unit zuul.service cannot be reloaded because it is inactive.
[21:01:14] <hashar>	 hmm
[21:01:19] <hashar>	 puppet still tries to reload it ahah
[21:01:37] <hashar>	 well anyway, it looks like it is good enough probably
[21:01:46] <mutante>	 it's better than before :p
[21:02:16] <mutante>	 sudo systemctl reload zuul
[21:02:16] <mutante>	 Failed to reload zuul.service: Unit zuul.service is masked.
[21:02:23] <mutante>	 this is also more obvious now
[21:02:42] <hashar>	 yeah I guess it is solved
[21:02:56] <mutante>	 just that list-units --all --state=masked somehow doesnt know this
[21:03:03] <mutante>	 and it does if you manually run the mask command
[21:04:08] <mutante>	 and even though all that does is create that symlink to /dev/null.. which exists
[21:04:49] <hashar>	 mutante: yeah I marked the task solved
[21:04:54] <hashar>	 and 
[21:05:00] <hashar>	 systemd::mask   should be removed
[21:05:13] <hashar>	 I might remember about it monday ;-]
[21:05:18] <hashar>	 thank you very much mutante !
[21:05:27] <mutante>	 hashar: ok, you're welcome. i will go for late lunch
[21:08:18] <hashar>	 mutante: happy lunch and good week-end!
[22:39:55] <wikibugs>	 10serviceops, 10Operations, 10HHVM, 10Patch-For-Review: Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10Dzahn) I uploaded a patch to allow us to clean hosts from hhvm resources by setting a Hiera key to absent. The example change applies it on mwdebug1001.eqiad.wmnet.  We can use...
[22:40:32] <wikibugs>	 10serviceops, 10Operations, 10HHVM, 10Patch-For-Review, 10Performance-Team (Radar): Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10Krinkle)
[22:59:53] <wikibugs>	 10serviceops, 10Operations, 10Release Pipeline, 10local-charts, and 3 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Jdforrester-WMF) Done?
[23:00:59] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 10Kubernetes, 10User-fsero: add CI job into operations/deployments-charts repo that helm lint packages and perform the helm index after merge. - https://phabricator.wikimedia.org/T216049 (10Jdforrester-WMF) 05Open→...
[23:01:01] <wikibugs>	 10serviceops, 10Operations, 10Release Pipeline, 10local-charts, and 3 others: Set up CI for the deployment-charts repository - https://phabricator.wikimedia.org/T233291 (10Jdforrester-WMF)