[09:11:47] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Proton, and 2 others: PDF download generates invalid PDF files - https://phabricator.wikimedia.org/T266559 (10Jgiannelos) @LGoto After some debugging on T266373 it looks like the issue is more complicated and affects more users than we a... [09:19:42] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Proton, and 2 others: PDF download generates invalid PDF files - https://phabricator.wikimedia.org/T266559 (10Jgiannelos) @sdkim Should i bump this to high priority? According to https://phabricator.wikimedia.org/T266373#6608804 there is... [10:37:18] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) [10:37:30] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) [11:07:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Build calico 3.16 - https://phabricator.wikimedia.org/T266893 (10JMeybohm) a:03JMeybohm I've crafted a debian package similar to how the k8s packages are not build (packages calico-cni and calicoctl as we currently have). Also I did crea... [11:21:37] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Archive/Remove deprecated calico gerrit repositories - https://phabricator.wikimedia.org/T267539 (10JMeybohm) [11:21:46] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Archive/Remove deprecated calico gerrit repositories - https://phabricator.wikimedia.org/T267539 (10JMeybohm) p:05Triage→03Low [11:25:25] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Archive/Remove deprecated calico gerrit repositories - https://phabricator.wikimedia.org/T267539 (10Aklapper) FYI https://phabricator.wikimedia.org/project/profile/2829/ links to https://phabricator.wikimedia.org/maniphest/task/edit/form/33/ cove... [12:45:29] 10serviceops, 10Performance-Team, 10User-jijiki: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10Gilles) [13:46:14] 10serviceops, 10Desktop Improvements, 10Operations, 10Product-Infrastructure-Team-Backlog, and 3 others: Connection closed while downloading PDF of articles - https://phabricator.wikimedia.org/T266373 (10Jgiannelos) @akosiaris More from debugging on this issue: * Querying directly the RESTBase service for... [13:46:50] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10Trizek-WMF) Thank you for the review and this list. I asked volunteers for translations and I will message the most impacted wikis on Fr... [13:47:51] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10Trizek-WMF) [14:22:26] apergos: are there are at least some kind of tests we could do on snapshot07 [14:22:31] that will not write on the db ? [14:22:37] for the icu update ? [14:23:22] we can't run the dump scripts, unfortunately, because I cannot guarantee no writes [14:23:40] I mean I can run one that just does a mysqldump but that's no help [14:23:44] moritzm ^ [14:23:52] ack, ok [14:24:35] apergos: you make us cry a bit [14:24:41] but hopefully we will recover [14:24:49] :D [14:25:10] sorry :-( [14:25:35] I would like to fix this someday so that we can run dumps as a separate user with no grants for write, that would be lovely [14:25:48] but fixing it is fixing some tings ni the bowels of mw core, it will be hard [14:25:53] *things in [14:26:27] I'll add to my todo list "see what was needed last time for testing in beta" [14:26:37] if indeed testing was done there ahead of time [14:28:17] 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10calbon) @akosiaris Can you review it? I don't know enough about the nodes vs redis connection to intelligently review. [14:28:38] apergos: thank you, we appreciate the good intent [14:29:50] sorry -( [14:29:52] :-( [14:30:36] _joe_: does https://gerrit.wikimedia.org/r/c/operations/puppet/+/639771 look reasonable to you? [14:35:00] <_joe_> cdanis: gerrit is down, but else, I had comments :) [14:35:25] really? works for me [14:35:29] <_joe_> ok, finally got through [14:35:41] <_joe_> cdanis: I pushed "send" while hashar was restarting gerrit :P [14:35:52] sorry :-\ [14:36:05] <_joe_> hashar: no problem, our timing was impeccable [14:36:12] really I should have scheduled some maintenance window early in the morning, but it typically takes less than a minute to come back [14:41:17] 10serviceops, 10Operations, 10CommRel-Specialists-Support (Oct-Dec-2020), 10User-notice: CommRel support for ICU 63 upgrade - https://phabricator.wikimedia.org/T267145 (10Trizek-WMF) I preventively [[ https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Update_to_ICU_Unicode_library | posted a... [15:59:24] may be a couple minutes late, onfire meeting may run over [15:59:35] but I'm coming, feel free to send me angry pings anyway :) [16:06:09] I've cleared the updateable packages on mwdebug2002, i.e. everything that is showing up after enabling the icu63 component will be related to the ICU update and in need to be included in the one "cumin apt-get install to rule them all" [16:07:00] thank you moritz, we have the list of the packages we use for the php upgrades [16:07:01] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: replace production deployment servers - https://phabricator.wikimedia.org/T265963 (10Papaul) [16:14:37] 10serviceops, 10Platform Engineering, 10observability, 10Developer Productivity: Set ENV SERVERGROUP for jobrunner MW web requests - https://phabricator.wikimedia.org/T266515 (10lmata) Please let us know if there is anything we can assist with... moving to radar meanwhile. [17:14:35] 10serviceops, 10Operations, 10Performance-Team, 10Platform Engineering: Phasing out MediaWiki Redis - https://phabricator.wikimedia.org/T267581 (10jijiki) [17:16:55] 10serviceops, 10Operations, 10Performance-Team, 10Platform Engineering: Phasing out MediaWiki Redis - https://phabricator.wikimedia.org/T267581 (10jijiki) [17:18:32] 10serviceops, 10Operations, 10Performance-Team, 10Platform Engineering: Phasing out MediaWiki Redis - https://phabricator.wikimedia.org/T267581 (10jijiki) [17:18:35] 10serviceops, 10Operations, 10Platform Engineering, 10User-jijiki: Upgrade MediaWiki's Redis cluster to Debian Buster - https://phabricator.wikimedia.org/T265643 (10jijiki) [18:22:05] 10serviceops, 10DBA, 10Operations: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10jijiki) We run the following on mwdebug2002: ` # 'export DEBIAN_FRONTEND=noninteractive; apt-get -y libicu63 libxml2' # 'export DEBIAN_FRONTEND=noninteractive; apt-get install php7.2-bcm... [18:27:32] 10serviceops, 10DBA, 10Operations: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10MoritzMuehlenhoff) That's not enough, these are just the binary packages from the php72 source package, but you also need to upgrade php-apcu, php-cli, php-common, php-excimer, php-geoip,... [18:30:54] 10serviceops, 10DBA, 10Operations: Upgrade the MediaWiki servers to ICU 63 - https://phabricator.wikimedia.org/T264991 (10jijiki) done [19:39:12] 10serviceops, 10Platform Engineering Roadmap Decision Making, 10Code-Health-Objective, 10Performance-Team (Radar), and 4 others: Determine multi-dc strategy for CentralAuth - https://phabricator.wikimedia.org/T267270 (10Gilles) [19:41:11] 10serviceops, 10Operations, 10Platform Engineering, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster - https://phabricator.wikimedia.org/T267581 (10Gilles) [19:41:19] 10serviceops, 10Operations, 10Platform Engineering, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster - https://phabricator.wikimedia.org/T267581 (10Krinkle) [19:44:09] 10serviceops, 10Operations, 10Platform Engineering, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster - https://phabricator.wikimedia.org/T267581 (10Krinkle) Clarified title and summary based on chat with Effie on IRC. The two remaining uses for a simpler non-replicated are Chrono... [20:52:20] 10serviceops, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) There are a couple of things that we will need to keep in mind: * **mcrouter... [20:56:20] 10serviceops, 10Operations: upgrade mwmaint1002 to buster - https://phabricator.wikimedia.org/T267607 (10jijiki) [20:56:46] 10serviceops, 10Operations: upgrade mwmaint1002 to buster - https://phabricator.wikimedia.org/T267607 (10jijiki) [20:56:52] 10serviceops, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Release-Engineering-Team (Deployment services): Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [21:16:02] 10serviceops, 10Operations, 10Platform Engineering, 10Performance-Team (Radar): Phasing out "redis_sessions" MediaWiki cluster - https://phabricator.wikimedia.org/T267581 (10jijiki) >>! In T267581#6614385, @Krinkle wrote: > Clarified title and summary based on chat with Effie on IRC. > > The two remaining... [21:51:24] 10serviceops, 10Packaging, 10cloud-services-team (Kanban): Package mcrouter 0.41 for Debian Buster - https://phabricator.wikimedia.org/T251574 (10jijiki)