[07:51:50] the full team meet is async today. are we doing the subteam meet? I have no opinion one way or the other, effie _joe_ akosiaris (won't ping mu tante because sleeping) [07:52:24] actually mu tante holiday I think, US tz [07:52:37] <_joe_> apergos: it would be a chance to meet our new team member if he comes online today (I think that's the case) [07:52:56] oh, is that the 14th? [07:53:04] * apergos checks notes [07:55:23] bah must be in the old notebook (cooincidentally, ordered the reMarkable today, so fed up) [07:55:29] no I think it is on Wed [07:55:31] day 1 [07:55:48] but I am not sure, if that is the case, I will agree with joe [07:56:07] I thought it was tomorrow :-D [08:01:07] <_joe_> so he starts this week, so today. I'm not sure if he'll be online given it's a WMF US holiday [08:01:17] <_joe_> but we will know in a few hours I guess [08:02:47] he won't have a google account for hangout yet I guess [08:03:33] he does have a wmf email so hmm [08:08:24] 10serviceops, 10Operations, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10Joe) [08:10:35] 10serviceops, 10Operations, 10Kubernetes: Upgrade the envoyproxy package to its latest version. - https://phabricator.wikimedia.org/T235412 (10Joe) [08:10:47] 10serviceops, 10Operations, 10Kubernetes: Upgrade the envoyproxy package to its latest version. - https://phabricator.wikimedia.org/T235412 (10Joe) p:05Triage→03High a:03Joe [08:24:53] _joe_: no he doesn't start today, he starts tomorrow [08:25:23] and we can do the team meeting async as well I guess [08:25:24] <_joe_> akosiaris: as I said, I'm not sure about that, but we will know later [08:25:59] I am [08:26:04] just read joel's email [08:26:20] quote: "His start date is Oct 15, 2019" [08:47:44] 10serviceops, 10Operations, 10Performance-Team, 10media-storage, and 2 others: Swift object servers become briefly unresponsive on a regular basis - https://phabricator.wikimedia.org/T226373 (10Gilles) Yes: ` gilles@ms-fe1005:/var/log/swift$ cat server.log.1 | grep ConnectionTimeout | wc -l 2048 ` [09:01:42] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10mobrovac) [11:46:35] 10serviceops, 10Operations, 10Performance-Team: webperf* running out of disk space - https://phabricator.wikimedia.org/T235425 (10jijiki) [12:16:13] 10serviceops, 10Operations, 10Performance-Team: webperf* running out of disk space - https://phabricator.wikimedia.org/T235425 (10Gilles) p:05Triage→03High [13:01:29] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Joe) >>! In T233654#5569671, @mobrovac wrote: > We have to resolve the same problem here to the one we encountered in Beta. Namely, both php-fpm and parsoid services use... [13:12:25] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10mobrovac) Aaah indeed, you are right @Joe. Not sure where I was looking :/ Must have been logged into a Beta instance. Ok, we don't need to do this whole port-switching... [14:10:41] 10serviceops, 10Citoid, 10Operations, 10Release Pipeline, 10Services: Migrate citoid and zotero services to helm ( scap-helm is deprecated ) - https://phabricator.wikimedia.org/T233702 (10jijiki) p:05Triage→03Normal [14:38:40] 10serviceops, 10Operations, 10Services, 10CPT Initiatives (RESTBase Split (CDP2)), and 2 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10akosiaris) p:05Triage→03High [14:39:13] 10serviceops, 10Operations, 10Services, 10CPT Initiatives (RESTBase Split (CDP2)), and 2 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10akosiaris) [14:39:20] 10serviceops, 10Operations, 10Release Pipeline, 10CPT Initiatives (RESTBase Split (CDP2)), and 4 others: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes - https://phabricator.wikimedia.org/T223953 (10akosiaris) [14:41:34] 10serviceops, 10Operations, 10Services, 10CPT Initiatives (RESTBase Split (CDP2)), and 2 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10akosiaris) For what is worth, the poolcounter approach is probably the saner one long term. And per https://... [14:41:40] 10serviceops, 10Operations, 10Release Pipeline, 10CPT Initiatives (RESTBase Split (CDP2)), and 4 others: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes - https://phabricator.wikimedia.org/T223953 (10akosiaris) In the interest of splitting off from this task what is probably going to be so... [15:11:03] 10serviceops, 10MediaWiki-REST-API, 10Wikimedia-Apache-configuration, 10CPT Initiatives (Core REST API in PHP): MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:11:32] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) Tagging #techcom for visibility. [15:16:54] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:18:17] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:19:54] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [15:23:55] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) According to Apache2 documentation, setting [`AllowEncodedSlashes NoDecode`](https://httpd.apache.or... [15:47:49] 10serviceops, 10Operations, 10Performance-Team: webperf* running out of disk space - https://phabricator.wikimedia.org/T235425 (10Krinkle) My guesses: * higher sampling rate from php7-excimer (compared to hhvm/xenon) is creating significantly larger files than we projected for in T199853. * general increase... [15:48:28] 10serviceops, 10Operations, 10Performance-Team: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Krinkle) [15:48:31] 10serviceops, 10Operations, 10Release-Engineering-Team, 10Scap, 10Wikimedia-General-or-Unknown: "Currently active MediaWiki versions:" broken on noc/conf - https://phabricator.wikimedia.org/T235338 (10jijiki) p:05Triage→03Normal a:03jijiki [15:49:00] 10serviceops, 10Operations, 10Release-Engineering-Team, 10Scap, and 2 others: "Currently active MediaWiki versions:" broken on noc/conf - https://phabricator.wikimedia.org/T235338 (10jijiki) [15:49:50] 10serviceops, 10Operations, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10jijiki) p:05Triage→03Normal [15:50:14] 10serviceops, 10Operations: Increase of varnish-be failed fetches error due to "http format error" - https://phabricator.wikimedia.org/T235254 (10jijiki) p:05Triage→03Normal [17:41:52] 10serviceops, 10Operations, 10Performance-Team: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Dzahn) looking at them now i see they are only using 14% and 8% of / . I ran "apt-get clean" and now it's down to 12% and 6%. Alerting would be at 95% by defa... [17:44:49] 10serviceops, 10Operations, 10Performance-Team: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10elukey) I noticed the warning this morning as well and pinged Gilles on IRC, the issue seems to be in the /srv partition: ` /dev/vdb 147G 135G 4.4G... [17:53:38] 10serviceops, 10Operations, 10Performance-Team: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Dzahn) sorry, i was on 1001 and 2001 vs. 1002 and 2002 and was wondering why i don't even see /srv mounted on a separate device. yes, ACK. on 1002 / 2002 it... [17:54:05] 10serviceops, 10MediaWiki-REST-API, 10TechCom, 10Wikimedia-Apache-configuration, and 2 others: MW REST API, wiki titles, Apache2 and URI decoding fun - https://phabricator.wikimedia.org/T235439 (10mobrovac) [18:10:13] 10serviceops, 10Operations, 10Traffic, 10Puppet: Puppet systemd::mask is an anti pattern that has unwanted side effect - https://phabricator.wikimedia.org/T233839 (10ema) 05Open→03Stalled [18:37:51] 10serviceops, 10Operations, 10CPT Initiatives (RESTBase Split (CDP2)), 10Core Platform Team Workboards (Clinic Duty Team), and 4 others: RESTBase/RESTRouter/service-runner rate limiting plans - https://phabricator.wikimedia.org/T235437 (10mobrovac) a:03mobrovac [19:43:11] 10serviceops, 10Operations, 10Performance-Team: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Krinkle) @dzahn These are not typical "log" files, they are active application data for arc lamp, public on perf.wm.o and consumed or reacted to by automated... [21:49:07] 10serviceops, 10Arc-Lamp, 10Operations, 10Performance-Team, 10Patch-For-Review: webperf*002 running out of disk space (arc lamp, xhgui) - https://phabricator.wikimedia.org/T235425 (10Krinkle) [22:30:39] 10serviceops, 10Arc-Lamp, 10Performance-Team: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10Krinkle)