[07:33:08] 10serviceops, 10Operations, 10WMDE-QWERTY-Team, 10wikidiff2, 10WMDE-QWERTY-Sprint-2019-05-15: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed - https://phabricator.wikimedia.org/T223391 (10MoritzMuehlenhoff) [08:07:29] 10serviceops, 10Operations, 10Patch-For-Review, 10User-Elukey: Renew certs for mcrouter on all application servers. - https://phabricator.wikimedia.org/T221346 (10fsero) i tried to reproduce the procedure outlined by @Joe and it won't work as cergen will create a new CA and new certificates. We can rollout... [08:18:38] 10serviceops, 10Operations, 10Patch-For-Review, 10User-Elukey: Renew certs for mcrouter on all application servers. - https://phabricator.wikimedia.org/T221346 (10fsero) actually cergen does it, it was a mistake on my part. [10:31:58] apergos, mutante it is just us 3 for today's status meeting [10:32:07] that's it? [10:32:09] yeah [10:32:18] we can skip it I think [10:32:57] where's f sero and a kosiaris? [10:33:10] not available [10:33:48] ok, well then let's update async (if $someone prepares the pad) and leave it at that [10:34:01] the pad is always there [10:34:47] I can do the team update [10:35:09] so it would be better if we just add 1-2 things we want to communicate to others [10:35:25] ok ? [10:45:39] oh, for the update out of the team, um [10:46:06] is there a separate section for that? because we (each of us) should just add an item or two if we have them, to that section [10:46:21] with no obligation that everyone has an item necessarily [10:46:35] does that make sense? (so no one person has to collect the items from someone else) [10:53:57] apergos: youc an add it on our weekly pad [10:54:03] and I can move it to the sre meeting one [10:54:05] no problem [10:54:12] nice [10:54:54] I have to leave halfway of the meeting, but I trust our turn will come before that [10:58:13] akosiaris, fsero please let us know if you are available after all [10:58:20] no i wont [10:58:25] i said no already to the meeting [10:58:25] so to officially cancel today's meeting [10:58:36] fsero: yeah I am summing up :p [10:58:43] I didn't check the list [10:58:45] you are right [10:59:39] fsero: sigh, my calendar doesn't show you declined it [10:59:42] anyway [10:59:42] also im on holidays next week so neither next meeting [10:59:43] :P [11:08:36] 10serviceops, 10Operations, 10User-jijiki: Investigate increase in GET ops registered by mcrouter for the mediawiki appserver cluster - https://phabricator.wikimedia.org/T223647 (10jijiki) [12:18:00] 10serviceops, 10Operations, 10Patch-For-Review, 10User-Elukey: Renew certs for mcrouter on all application servers. - https://phabricator.wikimedia.org/T221346 (10fsero) 05Open→03Resolved besides minor cleanups this is done [15:31:23] hi? [15:32:59] mutante: your team left you alone :) (see backlog) [15:33:10] around 10:31 UTC [15:33:15] volans: i see:) thanks, just found out [15:44:22] mutante: that's volans bad wording we didnt left you alone it just happened that we cannot attend the meeting [15:44:27] :p [15:45:17] mutante: I may be off the meeting by the time we do team update [15:45:30] can you read ariel's update please? [15:45:51] volans: I will be here as well [15:47:28] :) [16:03:39] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10elukey) [16:11:24] jijiki: ACK, got it and added something [16:13:00] mutante: you may have to jus read the php7 updates as well :p [16:13:12] yes [16:13:16] i will [16:13:29] rmeoved beta feature and we migrated 2 async jobs to php7 [16:13:39] thanks! [16:14:53] ok [16:17:29] the mc1033 bandwidth ticket doesnt seem to have serviceops though? [16:18:10] are we asking the rest of the team to keep an eye of that? [16:18:15] on [16:23:52] jijiki: does wikitech count as an "appserver" - if so then it can be mentioned in the goal section [16:24:02] but doesnt matter [16:24:16] mmm I am not sure [16:24:22] you can do it on team updates though [16:24:24] ok [20:10:37] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Gilles) [20:12:35] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) Given we don't have past data to compare to, the raw key hits doesn't necessarily point to the cause. Chr... [20:22:24] 10serviceops, 10Operations, 10observability, 10PHP 7.2 support, and 2 others: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm. - https://phabricator.wikimedia.org/T223336 (10Gilles) [20:28:12] 10serviceops, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Patch-For-Review, and 2 others: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10jijiki) [20:38:16] 10serviceops, 10Operations, 10Kubernetes, 10User-fsero, 10User-jijiki: Upgrade calico in production to version 2.4+ - https://phabricator.wikimedia.org/T207804 (10jijiki) [20:58:44] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10elukey) @Krinkle the graph of ops/second clearly shows a bump from ~3k ops to ~12k ops, and the amount of gets to... [21:20:00] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar), 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) Thanks, I didn't see the op/s stat. That correlation seems fair indeed. I wonder how that would have regr... [21:22:38] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) [21:23:13] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) (Moving back from Radar per @elukey's comment showing pretty clear correlation that would make it impossible for... [22:05:40] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) [22:07:36] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) >>! From the **Task description**: >spiked on 2019-05-02 19:26 UTC. >{F29166140} From [SAL](https://tools.wmflab... [22:15:34] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Performance-Team, 10User-jijiki: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) @kostajh @Catrope Could you rule out whether it's probably that the SpecialHomepage may have caused an increase i... [22:18:40] 10serviceops, 10Growth-Team, 10MediaWiki-Cache, 10Operations, and 4 others: Investigate increase in tx bandwidth usage for mc1033 - https://phabricator.wikimedia.org/T223310 (10Krinkle) [23:35:20] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 5 others: Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10Krinkle) The official requirements of Stash are in li... [23:36:39] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 5 others: Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10Krinkle) [23:41:53] 10serviceops, 10MediaWiki-Cache, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 5 others: Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10Krinkle) Regarding Option 1: I do not know of any use...