[08:54:27] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10MoritzMuehlenhoff) mwdebug1003 is working just fine in my tests, both for various page browsing and edits. [10:20:26] 10serviceops, 10Commons, 10MediaWiki-File-management, 10MediaWiki-Uploading, and 2 others: 502 Server Hangup Error on esams for "Upload a new version of this file" on Special:Upload on Commons - https://phabricator.wikimedia.org/T247454 (10Aklapper) [11:16:00] 10serviceops: Create WMDE Namespace on docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T268984 (10Tarrow) [11:24:03] 10serviceops, 10Wikibase Release Strategy: Create WMDE Namespace on docker-registry.wikimedia.org - https://phabricator.wikimedia.org/T268984 (10WMDE-leszek) [12:07:47] 10serviceops, 10Performance-Team, 10User-jijiki: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10Gilles) It's now complete: https://github.com/gi11es/thumbor-docker Next I will experiment a little with [[ https://wikitech.wikimedia.org/wiki/Blubber | Blubbe... [15:05:48] 10serviceops, 10Operations, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10jijiki) [15:06:09] 10serviceops, 10Operations, 10cloud-services-team (Kanban): Upgrade labweb servers to buster - https://phabricator.wikimedia.org/T269004 (10jijiki) [15:06:13] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10jijiki) [17:12:14] 10serviceops, 10Operations, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17), 10Performance Issue, and 3 others: Strategy for storing parser output for "old revision" (Popular diffs and permalinks) - https://phabricator.wikimedia.org/T244058 (10Pchelolo) [17:38:39] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10jijiki) @MSantos I will let you know when it is ready, what is your deadline? It will help me with the scheduling a bit.... [20:28:24] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` deploy1002.eqiad.wmnet ` The log... [20:42:22] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` deploy2002.codfw.wmnet ` The log... [21:18:22] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['deploy2002.codfw.wmnet'] ` and were **ALL** successful. [21:55:58] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) [21:56:02] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10jijiki) That sounds lovely! Are there any sanity tests we could possibly do? Maybe @ssastry can give us an idea? [22:04:25] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) So I used httpbb to test it and we still have this issue, which looks like firewalling..to my surprise. {P13485} [22:17:00] 10serviceops, 10Operations, 10Parsoid: Upgrade Parsoid servers to buster - https://phabricator.wikimedia.org/T268524 (10Dzahn) @jijiki @rlazarus I had to manually restart ferm (meh! no alerts about that and should not happen) but that made the httpbb tests work now: ` [deploy1001:~] $ httpbb --hosts parse2... [22:22:14] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, and 2 others: Upgrade MediaWiki appservers to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) I used `httpbb` to run all the tests for appservers we have on mwdebug1003 and it reported no issues... [22:22:19] I ran all the httpbb appserver tests on mwdebug1003 and no issues ^ [22:22:39] also parse2001 now working on buster and passes base_url and "main" tests [22:23:03] ...except had to manually restart ferm (or reboot) after reimaging. [22:23:11] my mind was on parsoid specific requests [22:23:18] oh interesting [22:23:52] that's in addition. was just testing the appserver part of it, yes [22:24:00] but since it's both now... [22:24:24] yea, it's not ideal that the ferm restart was needed [22:24:42] but also rather that than a bigger problem [22:29:48] we should probably create a "tests_parsoid.yaml" with some assertions that are parsoid specific [22:30:47] regarding deployment servers on buster: among the issues there is "mysql-client" package is expected but needs to use mariadb .. will see what else [22:31:24] need to add checkboxes like on the appserver ticket [22:33:06] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) [22:34:30] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) [22:36:29] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10Dzahn) [23:08:24] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Replace production deployment servers and update them to Buster - https://phabricator.wikimedia.org/T265963 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['deploy1002.eqiad.wmnet'] ` Of which those **FAILED**: ` ['deploy1002.eqiad.wmn... [23:16:07] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Config, and 2 others: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Daimona) 05Open→03Invalid We can pull pcov from sury-php, so this shouldn'...