[09:26:18] 10serviceops, 10Prod-Kubernetes: Refactor users in production-images - https://phabricator.wikimedia.org/T274852 (10JMeybohm) [09:26:34] 10serviceops, 10Prod-Kubernetes: Refactor users in production-images - https://phabricator.wikimedia.org/T274852 (10JMeybohm) p:05Triage→03Low [10:30:20] 10serviceops, 10SRE, 10Patch-For-Review: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10elukey) @wkandek Hi! Do you think that we could find somebody in your team to work with me on this task? It seems very important and potentially blocking others (also the hosts are still... [11:45:03] 10serviceops, 10SRE, 10Patch-For-Review, 10User-jijiki: Upgrade memcached to version 1.6.x - https://phabricator.wikimedia.org/T270315 (10jijiki) [12:13:17] 10serviceops, 10Add-Link, 10GrowthExperiments-NewcomerTasks, 10Product-Infrastructure-Team-Backlog, and 3 others: Service operations setup for Add a Link project - https://phabricator.wikimedia.org/T258978 (10kostajh) 05Open→03Resolved I think this is done; we can open new tasks as needed. Thank you fo... [15:49:51] 10serviceops, 10MediaWiki-Cache, 10MediaWiki-General, 10Performance-Team, 10User-jijiki: Use monotonic clock instead of microtime() for perf measures in MW PHP - https://phabricator.wikimedia.org/T245464 (10Krinkle) [15:51:53] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Check/Rebuild all docker-pkg build docker images running on kubernetes - https://phabricator.wikimedia.org/T274254 (10JMeybohm) [17:09:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:10:31] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:11:15] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [18:30:06] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [18:31:44] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1347.eqiad.wmnet'] ` an... [18:33:18] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1352.eqiad.wmnet'] ` an... [18:35:24] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1346.eqiad.wmnet'] ` an... [19:01:53] 10serviceops, 10DC-Ops, 10ops-eqiad: (Need By: TBD) rack/setup/install mc10[37-54].eqiad.wmnet - https://phabricator.wikimedia.org/T274925 (10RobH) [19:53:14] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10Krinkle) p:05Triage→03High [19:53:23] 10serviceops, 10Performance-Team, 10Platform Engineering, 10Wikimedia-Rdbms, and 4 others: Determine and implement multi-dc strategy for ChronologyProtector - https://phabricator.wikimedia.org/T254634 (10aaron) [19:58:28] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1288.eqiad.wmnet', 'mw12... [20:01:18] 10serviceops, 10WikimediaDebug, 10Patch-For-Review, 10Performance-Team (Radar): Convert mwdebug VMs to debian buster - https://phabricator.wikimedia.org/T274023 (10Krinkle) [20:01:27] 10serviceops, 10WikimediaDebug, 10Patch-For-Review, 10Performance-Team (Radar): Convert mwdebug VMs to debian buster - https://phabricator.wikimedia.org/T274023 (10Krinkle) [20:02:04] 10serviceops, 10Performance-Team, 10WikimediaDebug: Fetch mwdebug backend server list from noc.wikimedia.org - https://phabricator.wikimedia.org/T268167 (10Krinkle) 05Open→03Resolved [20:02:32] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Convert mwdebug VMs to debian buster - https://phabricator.wikimedia.org/T274023 (10Krinkle) [20:18:58] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Convert mwdebug VMs to debian buster - https://phabricator.wikimedia.org/T274023 (10Dzahn) >>! In T274023#6830418, @MoritzMuehlenhoff wrote: > I've set mwdebug1002 to "inactive" in conftool to unblock deployments. Thanks! mwdebug1002 has been re... [20:20:32] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10hashar) I havent touched to EasyTimeline / ploticus in age. The version up to Stretch uses gd for rendering and that finds the f... [20:22:07] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10Legoktm) This could also could be caused by the Shellbox deployment. [20:30:27] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10hashar) p:05High→03Unbreak! In logstash: `environment var GDFONTPATH not found`. That started today at roughly 13:00 UTC wit... [20:33:59] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10hashar) **Note:** beside the few traces above, I am not actively dealing with the issue, it is late here and I am heading to bed' [20:36:29] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10Legoktm) >>! In T274822#6835154, @hashar wrote: > In logstash: `environment var GDFONTPATH not found`. That started today at rou... [20:52:04] 10serviceops, 10EasyTimeline, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10Legoktm) Ah right, this was caused by https://gerrit.wikimedia.org/r/c/mediawiki/libs/Shellbox/+/662134 Instead of having a glo... [21:09:39] 10serviceops, 10EasyTimeline, 10Patch-For-Review, 10Regression: [EasyTimeline] No text included / no font rendered / displayed at all in PNG graph output - https://phabricator.wikimedia.org/T274822 (10Legoktm) p:05Unbreak!→03High I'm downgrading this to just "high" because 1) timeline is an unmaintaine... [21:12:56] ^ not related to buster migration [21:13:05] so I removed the serviceops tag [23:34:25] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:35:23] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia...