[09:07:34] on bastionhosts: [09:07:35] # Used by parsoid deployers [09:07:35] include ::profile::scap::dsh [09:07:48] wonder if that is still true [10:08:49] <_joe_> mutante: ask subbu later [10:09:37] ok, will do! [10:10:09] some logstash hosts have issues because old logstash hosts were decom/ed and removed DNS but not removed from Hiera/puppet ..:/ [10:10:35] so when ferm service was reloaded unrelatedly.. fail because it couldn't look up the names in DNS anymore [10:10:55] doesnt mean the firewall is open or anything.. but systemd alerts.meh [10:28:03] <_joe_> did you remove the old host? [10:30:26] i uploaded a change for it and pinged on the related decom ticket.. acked the icinga alerts, so far [10:33:39] <_joe_> what's the change? that needs to be merged quickly [10:33:42] <_joe_> I can review [10:34:25] https://gerrit.wikimedia.org/r/c/operations/puppet/+/499433 [10:34:42] i confirmed this did not mean lack of iptables rules [10:36:40] <_joe_> still [10:36:47] <_joe_> the new bastion can't reach the server [10:37:39] yea.. true [10:39:13] <_joe_> so it's not urgent, but it's better if we get to it [10:40:13] yea, i was thinking to wait for US time / herron [10:44:48] <_joe_> I just discussed this with Filippo, it's safe to perform as it won't restart elasticsearch [10:44:52] ok, i can also do that and first disable puppet on them [10:44:55] in a moment [10:44:59] <_joe_> let's try to do it that way, yes [10:45:02] ok [10:45:05] <_joe_> remember it's two roles :) [10:45:18] ah, yea [10:54:48] disabled on logstash*, merged changed, enabled on logstash1007, ran puppet.. recovery [12:59:21] 10serviceops, 10Operations, 10cloud-services-team, 10Core Platform Team Backlog (Watching / External), and 2 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki) [13:01:10] 10serviceops, 10Operations, 10cloud-services-team, 10Core Platform Team Backlog (Watching / External), and 2 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki) Adding cloud services team in the loop [13:05:02] 10serviceops, 10Operations, 10cloud-services-team, 10Core Platform Team Backlog (Watching / External), and 2 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10aborrero) >>! In T195392#5061736, @jijiki wrote: > Adding cloud services team in the loop labweb1... [14:18:09] hi -- there are two patches for service-runner services + logging pipeline which look good to me but I'd like another set of eyes from service ops also as an heads up [14:18:16] https://gerrit.wikimedia.org/r/c/operations/puppet/+/498872/ https://gerrit.wikimedia.org/r/c/operations/puppet/+/496813/ that is [14:18:31] could you take a look ? thanks! [14:43:46] _joe_: I found a good one with docker-pkg , the seed images are not pulled apparently [14:43:58] so one can end up building images based on some obsolete version of the seed image [14:43:59] <_joe_> hashar: uh indeed [14:44:10] <_joe_> open a task, will you? [14:44:15] which mlean that hmm CI container never got updated [14:44:18] or might have been [14:44:21] but by "mistake" [14:44:22] <_joe_> there is a specific reason for that [14:44:26] yeah I can imagine [14:44:29] <_joe_> no no they have been AIUI [14:44:48] anyway [14:45:00] I went adding a couple docker pull for those seed images https://gerrit.wikimedia.org/r/#/c/integration/config/+/496579/1/fabfile.py ;] [14:45:05] and yeah I am filliing a bug right now [14:48:36] https://phabricator.wikimedia.org/T219398 ;) [14:49:43] akosiaris: qq, any specific reason for suggesting /healthz instead of just e.g. /health for a GET health check route? [14:51:28] _joe_: so anyway, it is probably not a priority. I have simply adjusted our build script to always pull the two seed images we use [14:52:35] <_joe_> ottomata: google wants it [14:52:40] oh? [15:06:03] 10serviceops, 10Operations, 10cloud-services-team, 10Core Platform Team Backlog (Watching / External), and 2 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki) @aborrero I though that this is related to T218615, that's why I added you all :) [15:33:03] do we actually want to move the meeting tomorrow to the morning again? [15:33:13] since we're still on winter time in europe it's at a pretty reasonable time as far as I'm concerned ;) [15:34:54] fine for me [15:35:22] fine to keep at the normal, later time? [15:35:31] (well, one hour earlier because... :P) [15:53:35] either are fine. morning even better than afternoon [15:57:59] morning is better than the afternoon, here in EET but I don't have anything scheduled [16:10:09] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Mooeypoo) >>! In T218812#5060050, @Krinkle wrote: >>>! Task description: >> If a user adds an expiring item t... [16:50:52] ottomata: https://stackoverflow.com/questions/43380939/where-does-the-convention-of-using-healthz-for-application-health-checks-come-f [16:51:17] ottomata: just following the trend tbh. Most important thing is ofc that we are consistent [17:11:11] interessting danke [17:35:22] mark: +1 morning [18:10:39] 10serviceops, 10Operations, 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10jijiki) [18:14:07] 10serviceops, 10Operations, 10cloud-services-team, 10Core Platform Team Backlog (Watching / External), and 2 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10jijiki)