[03:41:03] serviceops folks, you should be aware of https://phabricator.wikimedia.org/T217715 [03:41:21] read queries against the k8s codfw prometheus are currently disabled [03:41:26] (it is still gathering data though) [07:23:17] _joe_: can we upgrade the scap version in labs / should we ask wmcs to do the merge in hiera labs.yaml? [07:23:49] <_joe_> mutante: frankly, people can pick their version via wikitech hiera [07:24:09] <_joe_> I don't care, but if you have a patch from someone, merge it if it installs the latest version [07:24:42] _joe_: yes, i have a patch that changes it to 3.9.2-1 for the global labs.yaml [07:24:56] <_joe_> go on then [07:25:00] ok, tx [08:09:23] i wonder if " apache::def { 'HHVM': } [08:09:23] [08:09:53] can be removed when converting from apache to httpd (nowadays) [08:10:50] digs into it [09:04:50] i switched noc.wikimedia.org to stop using old apache module now [09:04:59] it was possible since the appservers were also done [09:05:20] now i just have puppetmaster left i think [09:54:39] <_joe_> ok, can I help with that? [10:01:38] _joe_: well, you could review https://gerrit.wikimedia.org/r/c/operations/puppet/+/451821 but i already got a +1 from Herron and it compiles. i am not worried so much about prod just that it has some side-effect on labs puppetmasters in projects or something [10:02:11] and then there is the task of getting people to stop using "simplelamp" in cloud VPS but i started it at least.. made a ticket and identified the users [10:02:16] <_joe_> you should probably try to cherry-pick it in a project with a standalone puppetmaster [10:02:37] true.. yea. ok, i will do that [10:02:38] <_joe_> it's the easiest way to test it [10:03:27] summons paladox with the gerrit standalone puppetmaster [10:13:29] 10serviceops, 10Citoid: JSTOR is blocking citoid IPs - https://phabricator.wikimedia.org/T216456 (10Mvolz) >>! In T216456#5000567, @Joe wrote: > @Mvolz this means we can reconfigure citoid to use both proxies? I don't know, but I'll email you the IPs I sent them :). [10:14:47] Hi yes you can use the git puppet master mutante [10:15:03] Currently in class [10:18:41] thanks [11:22:41] please ack or decline the invite for the changed meeting time of tomorrow :) [13:11:24] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10jijiki) [13:12:22] fwiw we're doing that centrally for all hosts [13:12:29] I have spreadsheet formulas and stuff [13:12:36] * paravoid straightens his manager hat [13:16:40] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10ArielGlenn) snapshot1005,6,7 lease will expire Feb 2020; see T213758#4880703 [13:21:57] <_joe_> paravoid: I'm worried about "stuff" :D [13:22:50] what do you mean? [13:24:09] also there is a Netbox report for that fwiw (not super usable I know, but worth mentioning ;) ) [13:45:33] paravoid: we are just doing a small run ourselves, I used what cas and volans suggested [13:52:59] not sure if there is much point in this but ymmv :) [13:55:59] i just clicked that report link and wondered if the "Failed" icon on it is expected [13:56:16] i mean it shows a list of hosts but also that [13:57:11] ah, i guess the failure is "not all servers have purchase dates set so i failed to get that value for them: [14:02:47] no, a report is reported as failure if there is anything logged as error. For that specific report both older than 5y and missing purchase date are considered errors, while older than 4.5y is considered warning [14:05:09] ah! ok [14:33:51] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10Dzahn) cobalt and phab1002 have been purchased on 2015-12-04. But phab1002 is just a temp replacement for phab1001. But cobalt is affected, that's Gerrit prod. [15:12:32] is there a deployment-prep equivalent for the k8s environment? [15:14:16] yup there is. ottomata is your man on this as he is the latest user and has some patches for it already [15:15:17] but it's role::beta::docker_services [15:15:45] a puppet class that just uses docker to "simulate" kubernetes a bit [15:15:54] barely enough, but should do for now [15:23:51] akosiaris: so...what does this all look like from a deployment perspective? How does it fit (at all) with CI (or does it)? Is there a document somewhere that explains things? [15:24:10] akosiaris: basically, I have questions, but don't even know enough to properly formulate them :) [15:25:52] Is there an example of how to deploy something using `role::beta::docker_services`? [15:25:55] ottomata: ^^ [15:34:29] ok so [15:34:35] i was able to do it urandom [15:34:37] no docs [15:34:38] :) [15:34:48] but I just used horizon puppet interface with hiera there [15:34:49] will link you [15:35:46] https://horizon.wikimedia.org/project/instances/326156fa-29be-40e5-987c-bd0df5b61c74/ [15:35:53] Puppet Configuration taab [15:36:11] scrolkl aall the way down and you can see the hiera I applied [15:36:27] stuf for profile::docker::engine and profiel::docker::runner [15:36:43] then I applied the role::beta::docker_services class [15:36:56] and to my astonishment it worked! [15:44:35] ottomata: Error: Unable to retrieve details for instance "326156fa-29be-40e5-987c-bd0df5b61c74". [15:45:33] ah [15:45:39] ok dunno how linking works then [15:45:45] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10jijiki) [15:45:50] urandom: instance name is : deployment-eventgate-analytics-1 [15:45:54] nm [15:45:56] yeah [15:50:38] ottomata: I feel like I missing so much context needed to make sense of this [15:50:58] urandom: [15:50:59] :) [15:51:02] how can I help? [15:51:08] like, what is `profile::docker::runner::service_defs:` ? [15:51:09] here's what I know from reading the puppet [15:51:20] is that your services configuration? [15:51:39] partly yse [15:51:40] https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/docker/runner.pp [15:51:59] each key is the service name [15:52:07] and then the value is the app config [15:52:17] or sorry [15:52:24] so [15:52:30] eventgate-ci['config'] is the app config [15:52:49] the rest in eventgate-ci object is params to service::docker class [15:52:58] https://github.com/wikimedia/puppet/blob/production/modules/service/manifests/docker.pp [15:53:23] each entry in service_defs results in a service::docker define being declared [15:53:28] so in this case it will be [15:54:13] https://www.irccloud.com/pastebin/VrHurnnG/ [15:54:38] so you are using service_defs to programatically declare docker::service instances [15:54:43] does that make sense? [15:55:12] btw there's no k8s involved here [15:55:26] its just a way of running the docker image that would be deployed to k8s [15:55:28] so your image is wikimedia/eventgate-ci? [15:55:29] in docker-machine [15:55:30] yes [15:55:41] OK [15:55:42] (unfortunate repository naming) [15:56:42] ah yes, right. the service_defs keys must match the image name [15:58:43] OK, I can give this a shot [15:58:53] ottomata: thanks [15:59:03] ya can help if it gets weird :) [16:00:45] <_joe_> sorry I thought that puppet code was straightforward enough [16:00:52] <_joe_> but yes, we need to document the whole process [16:01:10] 10serviceops, 10User-jijiki: Improve Scap2 testing - https://phabricator.wikimedia.org/T216518 (10jijiki) p:05Triage→03Normal [16:02:31] _joe_: context, I guess [16:03:06] ottomata: is /etc/eventgate-ci/config.yaml baked into your image? [16:03:44] example configs yes [16:03:44] <_joe_> urandom: the shim I wrote will mount the config file inside your image at /etc/$image_name/config.yaml IIRC [16:03:54] aha [16:04:08] perfect [16:04:13] but in beta I use the services_defs[eventgate-ci][config] [16:04:13] for that [16:04:30] _joe_: puppet code looks like it uses it from $config param only [16:04:35] https://github.com/wikimedia/puppet/blob/production/modules/service/manifests/docker.pp#L49-L55 [16:05:22] <_joe_> ottomata: yeah I mean, you pass in a config and it will create a yaml config file [16:05:31] <_joe_> that will be mounted inside the image at runtime [16:07:54] aye ya [16:39:33] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10Dzahn) [17:21:02] <_joe_> https://github.com/uber/makisu interesting [17:25:27] <_joe_> also https://github.com/uber/kraken which is basically the "pull images via bittorrent" idea I had in my mind for some time [22:32:29] 10serviceops, 10TechCom-RFC: RfC: Standards for external services in the Wikimedia infrastructure. - https://phabricator.wikimedia.org/T208524 (10kchapman) Last Call extended by one week. Now ending at: March 13 11pm PST (March 14 7:00 UTC, 8:00 CET) [23:50:10] 10serviceops, 10TechCom-RFC: RfC: Standards for external services in the Wikimedia infrastructure. - https://phabricator.wikimedia.org/T208524 (10greg) Is the canonical location of the text here on task or the wiki?