[00:00:48] solved that too by copying it back from buster to stretch [00:01:29] upgrading scap on mw2250 - scap pull is rsyncing now :) [00:05:18] 10serviceops, 10Operations, 10ops-codfw, 10User-jijiki: Degraded RAID on mw2250 - https://phabricator.wikimedia.org/T226948 (10Dzahn) the above was after "19:50 < mutante> !log built new scap version 3.11.1-1 on boron, copied to install1002, imported package with reprepro, copied from stretch to jessie and... [00:06:40] 10serviceops, 10Scap, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907): 'scap pull' stopped working on appservers ? - https://phabricator.wikimedia.org/T228328 (10Dzahn) the above was after: 19:50 < mutante> !log built new scap version 3.11.1-1 on boron, copied to... [00:11:43] 10serviceops, 10Operations, 10ops-codfw, 10User-jijiki: Degraded RAID on mw2250 - https://phabricator.wikimedia.org/T226948 (10Dzahn) 05Stalled→03Resolved 20:08 <+icinga-wm> RECOVERY - PHP7 rendering on mw2250 is OK: HTTP OK: HTTP/1.1 200 OK - 327 bytes in 0.074 second response time 20:10 <+logmsgbot... [00:17:29] 10serviceops, 10Scap, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907): Deploy scap 3.11.1-1 - https://phabricator.wikimedia.org/T228482 (10Dzahn) built, published, deployed and tested on mw2250. just needs to be rolled out across the cluster with debdeploy now. [00:17:47] 10serviceops, 10Scap, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907): Deploy scap 3.11.1-1 - https://phabricator.wikimedia.org/T228482 (10Dzahn) a:03Dzahn [00:18:30] 10serviceops, 10Scap, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201907): 'scap pull' stopped working on appservers ? - https://phabricator.wikimedia.org/T228328 (10Dzahn) p:05High→03Normal the new scap version fixes this issue on mw2250. scap pull works there ag... [00:39:43] on mwmaint2001 - puppet warnings - wants to remove a bunch of log directories for maintenance jobs but fails because they are not empty [04:52:20] 10serviceops, 10Operations, 10Patch-For-Review, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10fsero) I did a complete pull of all images and tags of our... [04:52:29] 10serviceops, 10Operations, 10Patch-For-Review, 10Release-Engineering-Team-TODO (201907), 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10fsero) p:05High→03Normal [08:44:14] 10serviceops, 10Wikibase-Termbox-Iteration-20, 10Wikidata-Termbox-Iteration-19, 10Patch-For-Review: Create termbox release for test.wikidata.org - https://phabricator.wikimedia.org/T226814 (10akosiaris) p:05Triage→03High [08:47:32] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10akosiaris) 05Open→03Resolved a:03akosiaris Agreed with @Krenair, closing for now. [09:37:27] mutante: I 've rebased on top of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524476/. I think that fixes most errors (there is still one, but looks fixable) [09:46:35] 10serviceops, 10Thumbor, 10Performance-Team (Radar), 10User-jijiki: Terminate Thumbor with SSL - https://phabricator.wikimedia.org/T180696 (10jijiki) TLS on haproxy it is then:) [09:59:11] 10serviceops, 10Operations, 10ops-codfw: (OoW) restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10jijiki) @Eevans Shall we mark restbase2009 as inactive on conftool? [10:35:26] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Update open_nsfw-- for Wikimedia production deployment - https://phabricator.wikimedia.org/T225664 (10Tgr) How is this related to {T214201}? It seems unnecessary to do both. [12:48:55] just to check: helmfile applies to staging are OK today? I saw a few happen so I guess it is ok? [13:14:34] Ok [13:14:43] It's ok tarrow [13:28:52] cool! [17:06:21] akosiaris: woohoo. thanks a lot [17:39:17] amended one more time and jenkins finally likes it :) [19:06:20] fsero: should log output from k8s containers by making it to kibana? [20:01:49] 10serviceops, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Undeploy electron service from WMF production - https://phabricator.wikimedia.org/T226675 (10Pchelolo) >>! In T226675#5350326, @thcipriani wrote: > Guessing this means we can delete `deployment-pdfrender02` in beta? Tha... [20:03:05] 10serviceops, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Undeploy electron service from WMF production - https://phabricator.wikimedia.org/T226675 (10thcipriani) >>! In T226675#5350333, @Pchelolo wrote: >>>! In T226675#5350326, @thcipriani wrote: >> Guessing this means we can... [22:46:03] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Dzahn) now also phab2001 has been switched to php-fpm and worker . it matches... [23:58:18] 10serviceops, 10Operations, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10Aklapper)