[06:00:45] https://docs.google.com/document/d/1xeVvJ6KjFBkNjVspPbY_PwEDHC7XPi0J5p1SqUXcCl8/edit#heading=h.l9htumpyedyj [08:14:27] 10serviceops, 10Operations, 10RESTBase-API, 10TechCom, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10mobrovac) [08:14:33] 10serviceops, 10Operations, 10RESTBase, 10RESTBase-API, and 4 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10mobrovac) 05Stalled→03Open [08:14:45] 10serviceops, 10Operations, 10RESTBase, 10RESTBase-API, and 4 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10mobrovac) Last step: deployment [08:15:41] 10serviceops, 10Operations, 10RESTBase, 10RESTBase-API, and 3 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10mobrovac) [08:20:32] <_joe_> mark: our team meeting is at the same time as the IR meeting [08:50:38] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), 10Wikimedia-production-error: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10Joe) Ok, we got different takeaways from that ticket (that I did read in the past). Let'... [08:57:49] 10serviceops, 10Release-Engineering-Team, 10Continuous-Integration-Config, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Joe) I don't think we can really read a yaml file at runtime. As php has no way to share information between... [08:58:54] 10serviceops, 10Release-Engineering-Team, 10Continuous-Integration-Config, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Joe) we could even think of just keeping these files served statically by http from a centralized service, i... [09:57:55] 10serviceops, 10Operations, 10Release Pipeline, 10Core Platform Team (RESTBase Split (CDP2)), and 5 others: Deploy the RESTBase front-end service (RESTRouter) to Kubernetes - https://phabricator.wikimedia.org/T223953 (10mobrovac) [10:21:42] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Upgrade memcached for Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10MoritzMuehlenhoff) [10:37:55] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10MoritzMuehlenhoff) [10:38:34] 10serviceops, 10Operations, 10Patch-For-Review: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10MoritzMuehlenhoff) [10:53:20] ah :/ [10:53:37] we could do friday maybe [10:53:38] difficult too [10:56:15] akosiaris: Did you have a chance to look at if there is more you need from the WMDE side for T220402? :) I'm realising we took quite some time to address the most recent changes but if you think there are more on the horizon it would be super helpful for planning our upcoming tasks. Thanks! [10:58:54] so I have 2 goals for the meeting today: a) discuss the serviceops presentation for the summit, what we are gonna present and who are gonna prepare & present it [10:59:01] b) an annual planning update [11:01:48] * volans subscribe for the latter :) [11:11:23] <_joe_> mark: it's ok I can skip the IR meeting for once [11:11:33] <_joe_> also I had some tasks for it and I did nothing :( [11:19:55] well the IR meeting has stuff to do for dublin as well eh [11:23:29] 10serviceops, 10Operations: Migrate Failoid hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224559 (10MoritzMuehlenhoff) [11:26:02] 10serviceops, 10Operations: Migrate Zookeeper/etcd conf cluster in codfw to Stretch - https://phabricator.wikimedia.org/T224560 (10MoritzMuehlenhoff) [11:29:12] 10serviceops, 10Operations: Migrate Failoid hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224559 (10Volans) +1 on the naming and +1 on buster, they just have firewall rules, so should be pretty straightforward and easy to do. [11:32:47] 10serviceops, 10Operations, 10Traffic: Migrate Failoid hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224559 (10Volans) [12:27:12] I can attend part of the IR and join [12:27:57] but I will have to be gone by :15 [12:28:13] 10serviceops, 10ORES, 10Operations, 10Scoring-platform-team: Migrate ORES Redis servers to Stretch/Buster - https://phabricator.wikimedia.org/T224569 (10MoritzMuehlenhoff) [12:28:46] akosiaris: how much you think the registry will complain if we roll restart the redis servers? [12:47:15] 10serviceops, 10Operations: Migrate pool counters to Stretch/Buster - https://phabricator.wikimedia.org/T224572 (10MoritzMuehlenhoff) [12:56:23] 10serviceops, 10Operations, 10Kubernetes: Migrate Kubernetes etcd clusters to Stretch/Buster - https://phabricator.wikimedia.org/T224574 (10MoritzMuehlenhoff) [12:57:03] 10serviceops, 10Operations, 10WMDE-QWERTY-Team, 10wikidiff2, and 2 others: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed - https://phabricator.wikimedia.org/T223391 (10WMDE-Fisch) [13:05:04] jijiki: gone from what by :15? [13:05:11] service ops meeting always ends at :15 right [13:05:32] tbh we can possibly keep it short [13:14:50] 10serviceops, 10Operations, 10Kubernetes: Migrate etcd networking cluster to Stretch/Buster - https://phabricator.wikimedia.org/T224577 (10MoritzMuehlenhoff) [13:20:47] <_joe_> mark: the two meetings are exactly at the same time [13:20:56] <_joe_> 17:30-18:15 [13:23:32] 10serviceops, 10Operations, 10Wikimedia-Etherpad: Migrate etherpad1001 to Stretch/Buster - https://phabricator.wikimedia.org/T224580 (10MoritzMuehlenhoff) [13:34:59] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) [13:35:03] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Next), 10Services (next): Citoid container: Our config.yaml provided via Docker is unused? - https://phabricator.wikimedia.org/T223344 (10Krenair) 05Open→03Invalid Now that I think more about it, that's probably... [13:35:45] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) [13:35:48] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Watching / External), 10Services (next): Zotero container: Production is running candidate version, last production version is broken due to lack of ca-certificates package - https://phabricator.wikimedia.org/T223345 (... [13:36:34] 10serviceops, 10Editing-team, 10Beta-Cluster-reproducible, 10Core Platform Team Backlog (Watching / External), 10Services (next): Zotero container: Production is running candidate version, last production version is broken due to lack of ca-certificates package - https://phabricator.wikimedia.org/T223345 (... [13:39:27] 10serviceops, 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) If we're content to stick with simple Docker instances due to beta's relatively small scale, then I suggest... [13:46:14] hmm I thought the IR was at :00 like last week [13:47:25] :-( [13:47:37] i propose one of you two joins service ops and one the other? [13:48:03] <_joe_> jijiki: I will be useless in the IR meeting, I had done none of my homework :( [13:48:05] or we could push it slightly earlier and hope daniel can make it [13:48:26] I have already voluteered to do the IR meeting [13:48:38] i think 30 mins is enough for service ops so can do half an hour earlier to avoid overlap [13:48:41] but not sure about daniel [14:13:29] how about [14:13:30] we move it [14:13:34] if daniel can make it, great [14:13:38] if not, i will fill him in separately [14:16:55] so you mean at 15:00 UTC? [14:17:04] <_joe_> in 45 minutes, yes [14:18:25] yes [14:18:46] let me move now, please ack/decline [14:22:31] also see the email about the SRE summit joel just sent out :) [14:58:53] I will need some 2-3' to join the meeting please [14:59:04] start without me [15:56:58] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP Fatal Errors on mw1275 after deployment - https://phabricator.wikimedia.org/T222452 (10Krinkle) [15:57:12] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP Fatal Errors on mw1275 after deployment - https://phabricator.wikimedia.org/T222452 (10Krinkle) [16:02:03] 10serviceops, 10Release-Engineering-Team, 10Continuous-Integration-Config, 10Epic: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) >>! In T223602#5219845, @Joe wrote: > I don't think we can really read a yaml file at runti... [17:24:03] 10serviceops, 10Operations, 10wikitech.wikimedia.org, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10bd808) ` $ ssh labweb1001.wikimedia.org $ sql labswiki Fatal error: Uncaught RuntimeException: RedisConnectionPool requires a Redis client... [17:46:42] 10serviceops, 10Operations, 10wikitech.wikimedia.org, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10bd808) >>! In T223393#5221742, @bd808 wrote: > I'm going to poke around in puppet a bit and try to figure out what manifest we are missing... [19:32:32] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [20:07:49] <_joe_> bd808: the problem is not puppet [20:08:13] <_joe_> It's the packaged versions of php extensions, and it's by design more or less [20:08:22] <_joe_> I can take a look tomorrow morning [20:08:42] <_joe_> or you can go look at the versions of all the php* packages installed on a mw appserver [20:09:42] <_joe_> (yes, you also need igbinary and other things to be upgraded) [20:32:59] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab2001.codfw.wmnet'] ` Of which those *... [20:34:32] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [21:40:06] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['phab2001.codfw.wmnet'] ` Of which those *... [22:17:43] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for host... [23:31:29] 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) [23:34:11] 10serviceops, 10Operations, 10vm-requests: ganeti VM request - miscweb2001 - equivalent of krypton - https://phabricator.wikimedia.org/T224323 (10Dzahn) 05Open→03Resolved https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=miscweb2001 [23:34:13] 10serviceops, 10Operations, 10Patch-For-Review: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn)