[04:29:26] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) >>! In T260330#6467414, @dr0ptp4kt wrote: > I may misunderstand, but would Swift-awareness still be useful for things which... [05:10:03] 10serviceops, 10MW-on-K8s, 10Operations, 10Platform Team Workboards (Purple): Make Shellbox actually do streaming - https://phabricator.wikimedia.org/T268427 (10tstarling) [05:10:16] 10serviceops, 10MW-on-K8s, 10Operations, 10Platform Team Workboards (Purple): Make Shellbox actually do streaming - https://phabricator.wikimedia.org/T268427 (10tstarling) p:05Triage→03Low [06:41:38] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10Joe) >>! In T260330#6639403, @tstarling wrote: >>>! In T260330#6467414, @dr0ptp4kt wrote: >> I may misunderstand, but would Swift-awar... [07:43:53] 10serviceops, 10observability, 10Developer Productivity, 10Patch-For-Review, 10Platform Team Workboards (External Code Reviews): Set ENV SERVERGROUP for jobrunner MW web requests - https://phabricator.wikimedia.org/T266515 (10Joe) 05Open→03Resolved [07:47:33] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Refactor our helmfile.d dir structure for admin - https://phabricator.wikimedia.org/T268434 (10JMeybohm) [07:47:42] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Refactor our helmfile.d dir structure for admin - https://phabricator.wikimedia.org/T268434 (10JMeybohm) p:05Triage→03Medium [09:45:56] hnowlan: are you around ? [10:06:50] hey effie [10:15:42] heads up for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/641730 [10:16:11] ack [10:56:49] hnowlan: do you know any url we can hit to check if that works as expected? [11:00:36] yep! https://en.wikipedia.org/w/rest.php/v1/search/page?q=pizza&limit=1 looks okay to me [11:01:02] we have not rolled it out yet :p [11:01:20] valetin and I have a disagreement with varnish-fe [11:02:06] oh, heh [11:02:18] I was wondering, appservers hadn't shown any change in requests [11:05:57] hnowlan: question [11:06:51] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10Trizek-WMF) I regularly check on the wikis, but I've been offline between Wednesday evening and now (I'll check the wikis after posting... [11:07:07] I was hitting rest.php without any params and I was wondering if it is ok it returns 404, I mean if that is what we want [11:07:25] that does not affect what we are doing now at all [11:07:38] it is a philosophical question :p [11:08:14] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10dr0ptp4kt) Thanks @tstarling and @joe. It sounds like this isn't the right fit for Wikifunctions. I appreciate the advice! [11:08:54] I think in REST terms that makes sense, 404 is for when you can't map an operation to an object (although maybe there's a special case for the root) [11:11:44] ok cool, it is a philosophical issue indeed [11:12:05] we are rolling out, it is going to take some time as puppet runs run [11:13:58] cool [13:14:32] 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10CDanis) Zero NEL reports of `http.response.invalid.content_length_mismatch` on PDFs since 2020-11-18T13... [13:58:30] 10serviceops, 10Operations, 10Traffic, 10Platform Team Workboards (Green): MW REST API should be routed to api_appserver MW cluster - https://phabricator.wikimedia.org/T268043 (10Pchelolo) 05Open→03Resolved a:03Pchelolo Tutum, now it seems to work: ` curl -i https://en.wikipedia.org/w/rest.php/v1/pa... [14:01:19] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor calico deploy strategy - https://phabricator.wikimedia.org/T267653 (10JMeybohm) I did a bit of testing and it looks as if it is totally possible to switch `helmfile.d/admin` to use helm3 and get rid of tiller there... [14:14:33] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) >>! In T245757#6637932, @Dzahn wrote: > While the... [14:14:54] 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Emdosis) Headers and Footers are still missing and the font is still Liberation instead of Georgia.. He... [14:15:16] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) [14:16:25] 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10CDanis) @Emdosis This task relates to very low-level issues stopping the PDFs from being delivered at a... [14:17:32] 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 4 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Emdosis) @CDanis Ok. [14:19:21] https://gerrit.wikimedia.org/r/c/operations/puppet/+/643028 is the last missing bit for a buster-based app server and having puppet running on mwdebug1003 [18:10:05] 10serviceops, 10Add-Link, 10GrowthExperiments-NewcomerTasks, 10Operations, and 2 others: Service operations setup for Add a Link project - https://phabricator.wikimedia.org/T258978 (10MMiller_WMF) a:03kostajh [20:30:08] 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests, 10Patch-For-Review: Move testreduce away from scandium to a separate Buster Ganeti VM - https://phabricator.wikimedia.org/T257906 (10Dzahn) 05Resolved→03Open npm install is not working on testreduce1001 [20:31:58] 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests, 10Patch-For-Review: Move testreduce away from scandium to a separate Buster Ganeti VM - https://phabricator.wikimedia.org/T257906 (10Dzahn) Now there is the next issue: ` npm ERR! Error while executing: npm ERR! /usr/bin/git ls-remote -h -t https:... [20:32:14] 10serviceops, 10Operations, 10Parsoid, 10Parsoid-Tests, 10Patch-For-Review: Move testreduce away from scandium to a separate Buster Ganeti VM - https://phabricator.wikimedia.org/T257906 (10Dzahn) As reported by subbu, 'npm install' was not working on testreduce1001. It hung after "loadIdealTree:loadAllDe... [21:00:47] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) >>! In T245757#6640719, @MoritzMuehlenhoff wrote: >>>! In T245... [21:07:29] 10serviceops, 10Parsoid, 10Patch-For-Review: equivalent of scandium on buster? (upgrading parsoid testing servers) - https://phabricator.wikimedia.org/T268248 (10jijiki) >>! In T268248#6635584, @Dzahn wrote: > @jijiki @hnowlan @RLazarus Here's the summary. > > There is `scandium.eqiad.wmnet` which is a Par... [21:18:25] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) Using `dist => "${::lsbdistcodename}-wikimedia",` instead of e... [21:30:50] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [21:31:36] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [21:38:55] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) >>! In T245757#6640719, @MoritzMuehlenhoff wrote: > * I edited... [21:41:33] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) >>! In T245757#6642420, @Dzahn wrote: > Using `dist => "${::l... [21:58:45] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [21:58:50] 10serviceops, 10Performance-Team, 10WikimediaDebug, 10Patch-For-Review: create mwdebug1003 - ganeti VM with buster and appserver role - https://phabricator.wikimedia.org/T267248 (10Dzahn) 05Open→03Resolved VM has been created, added to Debug extension, added to conftool. Puppet role applied, puppet se... [21:59:11] 10serviceops, 10Performance-Team, 10WikimediaDebug: create mwdebug1003 - ganeti VM with buster and appserver role - https://phabricator.wikimedia.org/T267248 (10Dzahn) [22:18:14] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10RLazarus) >>! In T267145#6640181, @Trizek-WMF wrote: > @RLazarus, what was this hiccup you mention? People may like to know about the de... [22:48:51] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) @jijiki I see the "mcrouter 0.37" checkbox above. On mwdebug1... [22:52:51] puppet run on mwdebug1003 is now without any warning or error, and also not repeating things one each run for the first time. puppet installs correct APT component and all the packages. it all works :) [22:58:53] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) >>! In T245757#6640719, @MoritzMuehlenhoff wrote: > upgraded... [22:59:53] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) [23:10:37] 10serviceops, 10Parsoid, 10Patch-For-Review: equivalent of scandium on buster? (upgrading parsoid testing servers) - https://phabricator.wikimedia.org/T268248 (10Dzahn) >>! In T268248#6642370, @jijiki wrote: > If I am understanding correctly, the parsoid upgrade to buster has no dependencies on this task, co... [23:11:04] 10serviceops, 10Parsoid, 10Patch-For-Review: equivalent of scandium on buster? (upgrading parsoid testing servers) - https://phabricator.wikimedia.org/T268248 (10Dzahn) p:05Triage→03Medium