[07:51:30] Morning service ops people! Could anyone give me any hints about what needs to be done for https://phabricator.wikimedia.org/T226814? I'm happy to [07:51:38] do some of the bits that are needed [08:09:02] 10serviceops, 10Machine vision, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban), and 2 others: Update open_nsfw-- for Wikimedia production deployment - https://phabricator.wikimedia.org/T225664 (10Joe) >>! In T225664#5329541, @Mholloway wrote: > @joe Thanks (belatedly) for the comments. I've... [08:38:27] 10serviceops, 10MediaWiki-Logging, 10Operations, 10Wikimedia-Logstash, and 8 others: Port mediawiki/php/wmerrors to PHP7 and deploy - https://phabricator.wikimedia.org/T187147 (10Joe) >>! In T187147#5343202, @tstarling wrote: > Is this blocking deployment of PHP 7? In my opinion, it should not at this poi... [08:39:12] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10PHP 7.2 support, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) @thcipriani do you need more information from us? Any idea when work on scap will begin? [08:49:13] 10serviceops, 10Parsoid-PHP, 10Core Platform Team (Parsoid REST API in PHP (CDP2)): Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing - https://phabricator.wikimedia.org/T228069 (10Joe) [08:50:56] 10serviceops, 10Parsoid-PHP, 10Core Platform Team (Parsoid REST API in PHP (CDP2)): Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing - https://phabricator.wikimedia.org/T228069 (10Joe) Can I ask why such parser tests will need to run in production rather than via the CI infrastru... [09:04:12] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), and 2 others: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10Joe) Frankly without further information on what happened, how it was debugged, and how it was solved, I'm... [10:13:45] 10serviceops, 10Operations: SRE FY19-20 Q1 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10jijiki) [10:43:02] 10serviceops, 10Operations, 10PHP 7.2 support: Don't monitor HHVM on PHP7 only servers - https://phabricator.wikimedia.org/T228643 (10jijiki) [10:43:18] 10serviceops, 10Operations, 10PHP 7.2 support: Don't monitor HHVM on PHP7 only servers - https://phabricator.wikimedia.org/T228643 (10jijiki) [10:43:21] 10serviceops, 10Operations, 10Performance-Team (Radar), 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10jijiki) [10:43:29] 10serviceops, 10Operations, 10PHP 7.2 support: Don't monitor HHVM on PHP7 only servers - https://phabricator.wikimedia.org/T228643 (10jijiki) p:05Triage→03Normal a:03jijiki [12:13:28] 10serviceops, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Undeploy electron service from WMF production - https://phabricator.wikimedia.org/T226675 (10jijiki) 05Open→03Resolved [13:28:52] akosiaris: _joe_ coredns works in staging now https://www.irccloud.com/pastebin/KZnOCjzu/ [13:29:42] \o/ [13:59:06] <_joe_> \o/ [14:08:20] Oooh! Anyway we can use that to avoid having to rewrites from external DNS names to the internal app-server.discovery.wmnet name? Probably aiming it at an in cluster proxy container to handle TLS? [14:15:36] <_joe_> tarrow: no, that's not the plan. In a non-distant future, TLS will be completely handled for you, but via other means. [14:18:22] _joe_: ok, any plans to allow containers to be ignorant of the whole write rewrite DNS and add 'Host:' headers dance? Would be super awesome to have that handled by the infrastructure :) [14:20:43] <_joe_> tarrow: I'm not 100% sure what you mean, but [14:25:49] <_joe_> right now, you basically have to have your application connect to some address and then behave like requesting the domain you want to reach, like [14:25:49] Ah, I mean. At the moment if services want to talk to the application servers about enwiki they need to request 'appservers-ro.discovery.wmnet' and set 'Host:en.wikipedia.org'. Rather than just requesting en.wikipedia.org. Otherwise they go to the edge routers and back again [14:26:50] <_joe_> yeah that won't change, what will change is you will just have to connect, for all services, to a localhost address, via http (not https) [14:27:11] <_joe_> we could think of more radical solutions in the future, but I think that's reasonable [14:28:06] That means that we have to have a load of special extra code altering http requests that only comes into play when being on the WMF prod cluster [14:28:53] It could be pretty awesome if that was abstracted away by the orchestration so that I can run the same container with the same configuration either in the cluster or not and still have it work [14:31:17] I have the feeling I'm not explaining it very well. And sadly I now have to head off. Thanks for listening to my ramble though :) [14:33:07] <_joe_> no I think I got what you mean [14:33:53] <_joe_> and well, I think there is a solution (kind-of) for that. In helm chart terms. So that you can use the same code in local dev and in production [14:51:48] tarrow: re T226814 could you create a new release of termbox with a NodePort serving in a different port than production [14:51:49] like 3031? [15:05:51] akosiaris: _joe_ [15:06:04] <_joe_> ? [15:06:05] I may not attend our meeting [15:06:20] I will merge now the change to migrate some servers to php7 only [15:06:23] and I will monitor a bit [15:06:32] 10serviceops, 10Parsoid-PHP, 10Core Platform Team (Parsoid REST API in PHP (CDP2)): Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing - https://phabricator.wikimedia.org/T228069 (10ssastry) >>! In T228069#5352568, @Joe wrote: > Can I ask why such parser tests will need to run in p... [15:06:40] I may join depends of how it goes [15:07:45] ok [15:11:58] 10serviceops, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Core Platform Team Workboards (Clinic Duty Team), and 3 others: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10Pchelolo) [15:22:36] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Pipeline): Self-service Deployment Pipeline - https://phabricator.wikimedia.org/T228676 (10Jdforrester-WMF) [15:23:11] 10serviceops, 10Operations, 10Release Pipeline, 10Goal, 10Release-Engineering-Team (Pipeline): Self-service Deployment Pipeline - https://phabricator.wikimedia.org/T228676 (10akosiaris) p:05Triage→03Normal [15:33:49] 10serviceops, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Core Platform Team Workboards (Clinic Duty Team), and 4 others: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10WDoranWMF) [16:03:33] 10serviceops, 10Wikibase-Termbox-Iteration-20, 10Wikidata-Termbox-Iteration-19, 10Patch-For-Review: Create termbox release for test.wikidata.org - https://phabricator.wikimedia.org/T226814 (10WMDE-leszek) Thanks for the work and your thoughts on this so far @akosiaris and @fsero! Would you be able to estim... [16:28:39] fsero: yes! of course I can :) Doesn't need a new chart though right? just a new values.yaml? [16:29:23] no need chart, problably a new directory copied from termbox under services/staging [16:36:24] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), and 2 others: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10Krinkle) Logstash query for the error in question: fsero: cool; PS is up [16:44:20] 10serviceops, 10Operations, 10PHP 7.2 support, 10Performance-Team (Radar), and 2 others: PHP 7 corruption during deployment (was: PHP 7 fatals on mw1262) - https://phabricator.wikimedia.org/T224491 (10ArielGlenn) From IRC conversation, Krinle says he ran php7adm /opcache-free and the problem immediately... [16:54:23] I'll need more time tarrow thanks [17:03:38] fsero: right ho :) I'm done for the day anyway; just keeping up while I cook tea. No rush [17:16:48] hello people [17:17:04] https://phabricator.wikimedia.org/T226778 contains a lot of subtasks related to the PDU work for row a and row b [17:17:14] IIUC today a4 and a5 should be done [17:17:24] there are some hosts that might need some review [17:17:37] a4 https://phabricator.wikimedia.org/T227140 [17:17:50] a5 https://phabricator.wikimedia.org/T227141 [17:19:19] (conf1004, scb1001, kafka1001, oresrdb1002) [17:31:45] 10serviceops, 10Scap, 10PHP 7.2 support, 10Patch-For-Review, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5352559, @Joe wrote: > @thcipriani do you need more information from us? Any idea when work on scap will b... [17:36:32] depooled scb1001 FYI for the a4 work