[05:48:31] 10serviceops, 10ChangeProp, 10WMF-JobQueue, 10CPT Initiatives (Containerise Services): Make changeprop chart depend on Kafka-dev for minikube - https://phabricator.wikimedia.org/T245803 (10Pchelolo) [08:44:47] the docker-reporter-base-images unit on boron failed for docker-registry.wikimedia.org/python3:0.0.2, known issue? [09:46:41] <_joe_> no [10:39:38] moritzm: regarding mw1221-mw1258 and mw2135-mw2214, you can ask mutante and rlazarus when they are online [10:39:45] I believe it wont be long [10:39:51] or I will ping them later [10:40:51] ack [11:17:53] <_joe_> moritzm: it was docker failing to perform an operation [11:17:59] <_joe_> Feb 21 01:02:15 boron docker-report-base[17730]: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration for process caused \"failed to [11:18:36] ack [11:18:46] <_joe_> write 24042 to cgroup.procs: [11:19:23] 10serviceops, 10Analytics, 10Operations, 10vm-requests, 10Patch-For-Review: Create a ganeti VM in eqiad: an-launcher1001 - https://phabricator.wikimedia.org/T244717 (10elukey) [14:16:18] haha, it is that time of day again! [14:16:53] ottomata: you have had better opening lines, you are letting me down :/ [14:16:55] * ottomata finishes cup of coffee and finsihes checking emails and pings akosiaris about k8s namespaces [14:17:03] hahaha [14:17:09] ok I will work on it [14:17:21] haha [14:19:20] ottomata: at an interview [14:21:36] k! [15:20:34] <_joe_> ottomata: when you want to add a new service, where do you look for documentation? [15:23:07] _joe_: in k8s? [15:23:18] or in general? [15:23:26] i always start with wikitech [15:23:48] but at this point i've done it enought that I know all of the basics [15:23:55] <_joe_> ok [15:24:02] <_joe_> good news, you'll have to relearn it all! [15:24:05] haha [15:24:13] great! you mean with the helm stuff, or just the lVS stuff? [15:26:26] <_joe_> for now the LVS stuff [15:26:44] <_joe_> I'll wait until you've formed a proper muscle memory before changing the helm stuff too much [15:26:58] <_joe_> I want to properly frustrate the users :P [15:27:11] ah [15:27:15] <_joe_> jokes aside [15:27:23] haha, well i look forward to some helm changes, there is a lot of repetition currently [15:27:25] but for LVS [15:27:26] <_joe_> I don't think the helm stuff will change significantly [15:27:28] hm, yeah i check wikitech [15:27:33] then I look in puppet [15:27:41] i submitted patches this week [15:27:42] <_joe_> it's mostly fixing the repetitiveness in helmfile, yes [15:27:47] with some new stuff you've been doing [15:27:53] less repitition for LVS now for sure [15:27:56] are there new new changes? [15:28:12] this is the LVS patch I did this week [15:28:12] https://gerrit.wikimedia.org/r/c/operations/puppet/+/573365 [15:28:23] <_joe_> yeah, now we have the concept of service "state" [15:28:47] <_joe_> which allows to apply changes in the ccorrect order when we add a service to lvs [15:29:07] <_joe_> and hopefully also avoid paging [15:29:48] ah nice [15:29:59] i've never applied any changes myself, always left that to the pros [15:31:24] <_joe_> the idea is we won't need that anymore [15:31:39] <_joe_> I said my goal was you being able to do it without involving us [15:32:01] <_joe_> where you're a shortcut for "a random SRE not working on this day to day" [15:45:49] yeahhhhh [15:45:53] sounds amazing [15:46:23] _joe_: is that ready now? I'll want to do this LVS for eg-analytics-external hopefully early next week [15:46:32] if you write some docs, maybe I can try? [15:47:12] <_joe_> ottomata: no, so [15:47:26] <_joe_> it should be by monday [15:51:50] well monday is early next week [15:51:51] :) [15:57:08] 10serviceops, 10Analytics, 10Operations, 10vm-requests: Create a ganeti VM in eqiad: an-launcher1001 - https://phabricator.wikimedia.org/T244717 (10elukey) 05Stalled→03Open ` Creating new VM named an-launcher1001.eqiad.wmnet in eqiad with row=C vcpu=4 memory=8 gigabytes disk=100 gigabytes link=analytic... [16:21:02] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10jijiki) [16:22:01] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10jijiki) [16:42:16] 10serviceops, 10Analytics, 10Operations, 10vm-requests, 10Patch-For-Review: Create a ganeti VM in eqiad: an-launcher1001 - https://phabricator.wikimedia.org/T244717 (10elukey) 05Open→03Resolved a:03elukey [19:35:57] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) [19:39:28] 10serviceops, 10Operations, 10observability, 10Patch-For-Review: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) @herron any ideas how to proceed here? Is there someone who can help? [19:39:50] 10serviceops, 10Beta-Cluster-Infrastructure, 10Operations, 10observability: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) [19:55:50] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap: Deploy scap 3.13.0-1 - https://phabricator.wikimedia.org/T245530 (10jijiki) Package has been build and pushed to wikimedia repo, I'll roll it to production on Monday [20:08:21] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2366.codfw.wmnet ` The log can be found in `/var/log... [20:09:33] 10serviceops, 10Beta-Cluster-Infrastructure, 10Operations, 10observability: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10herron) Looking a bit closer I think this is happening because the nodes in labs are assigned their roles/profiles/etc via the exte... [20:22:08] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2367.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [20:34:24] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2366.codfw.wmnet'] ` and were **ALL** successful. [20:35:16] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2368.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [20:44:09] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2367.codfw.wmnet'] ` and were **ALL** successful. [20:45:24] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2369.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [20:57:10] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2368.codfw.wmnet'] ` and were **ALL** successful. [20:57:44] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2370.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [21:05:38] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2369.codfw.wmnet'] ` and were **ALL** successful. [21:06:24] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2371.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [21:19:48] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2370.codfw.wmnet'] ` and were **ALL** successful. [21:27:20] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2371.codfw.wmnet'] ` and were **ALL** successful. [21:28:51] 10serviceops, 10Beta-Cluster-Infrastructure, 10Operations, 10observability: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) I have uploaded a patch which I manually tried on beta, this seems to work, but sadly, puppet breaks a bit further down the... [21:49:12] 10serviceops, 10Beta-Cluster-Infrastructure, 10Operations, 10observability: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) With a little bit more fiddling, I managed to run puppet on ssh deployment-mediawiki-09.deployment-prep.eqiad.wmflabs! @her... [22:14:08] 10serviceops, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforrester-WMF) [22:18:15] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production deployment access - https://phabricator.wikimedia.org/T245877 (10Jdforrester-WMF) [22:19:31] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Jdforrester-WMF) [22:24:43] 10serviceops, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10cscott) [22:30:12] 10serviceops, 10Parsoid-PHP, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Create /srv/mediawiki/parsoid-vendor on production MW appservers, a check out of vendor.git's parsoid branch - https://phabricator.wikimedia.org/T245886 (10Jdforrester-WMF) [22:32:49] 10serviceops, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Krinkle) [22:34:20] 10serviceops, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10cscott) [22:37:08] 10serviceops, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10Jdforrester-WMF) [22:47:32] 10serviceops, 10Parsoid-PHP, 10MW-1.35-notes (1.35.0-wmf.20; 2020-02-18), 10Patch-For-Review: Craft a deployment strategy to transition Parsoid/PHP from a faux extension to a composer library without breaking incoming requests - https://phabricator.wikimedia.org/T240055 (10cscott) [23:16:00] 10serviceops, 10Core Platform Team, 10DC-Ops, 10Operations: Rename wtp* servers to parsoid* (Parsoid PHP servers) - https://phabricator.wikimedia.org/T245888 (10jijiki)