[00:44:15] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) The "use_php" parameter we had for the migration has been removed from puppet code now. All instances are using it by default. [06:44:00] chaomodus: I am off till January, I will be around only to finish a couple of things I didnt finish yesterday [06:44:03] sorry:/ [08:19:13] [08:48:04] 10serviceops, 10MediaWiki-Docker, 10Operations, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10kostajh) The logs don't seem to show that much useful information. I set "debug: true" in the daem... [09:24:25] 10serviceops, 10MediaWiki-Docker, 10Operations, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10JMeybohm) Unfortunately still unable to reproduce, even with docker & docker-engine 20.10.1 (on li... [09:40:28] 10serviceops, 10MediaWiki-Docker, 10Operations, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10kostajh) I can reproduce on Debian unstable with `docker pull docker-registry.wikimedia.org/dev/st... [10:06:52] FYI I've sent https://gerrit.wikimedia.org/r/c/operations/puppet/+/650092 for legoktm's onboarding [10:35:17] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10JMeybohm) I was able to reproduce as well now. Seems like we are missing the "content... [10:51:21] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10JMeybohm) p:05Low→03High Also rising priority as I guess this will affect more an... [14:03:41] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10jijiki) @MSantos Credentials are in place, everything should be in place if you deploy again. Let me know if there is so... [16:22:14] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2020.codfw.wmnet ` The log can be... [16:22:18] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1020.eqiad.wmnet ` The log can be... [16:31:54] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10JMeybohm) 05Open→03Resolved a:03JMeybohm For the record: We where sending "Content-Type:... [16:31:59] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) This should now be fixed by using https://gerrit.wikimedia.org/r/c/operations/puppet/+/650... [16:40:05] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10kostajh) 05Resolved→03Open >>! In T270270#6698851, @ema wrote: > This should now be fixed b... [16:41:39] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10JMeybohm) Maybe Mac sends a different user-agent? That would be fun... [16:59:44] 10serviceops, 10Operations, 10Traffic, 10HTTPS: puppetmaster[12]001: add TLS termination - https://phabricator.wikimedia.org/T263831 (10jbond) 05Open→03Resolved a:03jbond Sorry for the delay however this has been configured now [17:04:11] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) also see T108580 [17:04:47] 10serviceops, 10Operations, 10Traffic, 10HTTPS: puppetmaster[12]001: add TLS termination - https://phabricator.wikimedia.org/T263831 (10Dzahn) @jbond @ema So puppetmaster1001 can also be checked off on T210411 ? [17:09:36] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1020.eqiad.wmnet'] ` and were **ALL** successful. [17:11:14] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2020.codfw.wmnet'] ` and were **ALL** successful. [17:27:55] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) >>! In T270270#6698866, @JMeybohm wrote: > Maybe Mac sends a different user-agen... [17:31:56] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) [20:31:53] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, 10User-zeljkofilipin: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) The current theory is that the problem boils down to the following HEAD request... [21:09:05] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) >>! In T270270#6699691, @ema wrote: > That being said, I suspect that our VCL trying to do... [21:10:32] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10kostajh) >>! In T270270#6699794, @ema wrote: >>>! In T270270#6699761, @gerritbot wrote: >> Chan... [21:16:30] 10serviceops, 10MediaWiki-Docker, 10Operations, 10Traffic, and 2 others: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` - https://phabricator.wikimedia.org/T270270 (10ema) >>! In T270270#6699806, @gerritbot wrote: > Change 650191 had a related patch set uploaded... [21:57:29] 10serviceops, 10Graphoid, 10Operations, 10Platform Engineering (Icebox): Undeploy graphoid for phase 4 wiki's - https://phabricator.wikimedia.org/T270443 (10Jseddon) [23:37:44] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: eqiad: Physical moves for MediaWiki servers - https://phabricator.wikimedia.org/T266164 (10Dzahn) a:05Dzahn→03Cmjohnson Assigning to you to find out if we need to keep this open or not. If it's done, feel free to close as resolved...