[11:59:00] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [12:20:20] 10serviceops, 10Add-Link, 10Growth-Team: Add Link engineering: Puppetize DB credentials - https://phabricator.wikimedia.org/T269573 (10kostajh) [13:06:55] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10JMeybohm) [13:07:07] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10JMeybohm) 05Open→03Resolved [13:07:09] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10JMeybohm) [13:11:26] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10JMeybohm) [13:12:14] 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review, 10Release Pipeline (Blubber): Move blubberoid to use TLS only. - https://phabricator.wikimedia.org/T236017 (10JMeybohm) 05Open→03Resolved [13:12:40] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10JMeybohm) [13:13:01] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Add TLS termination to services running on kubernetes - https://phabricator.wikimedia.org/T235411 (10JMeybohm) [13:13:08] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10JMeybohm) 05Resolved→03Open [13:13:29] 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review, 10Release Pipeline (Blubber): Move blubberoid to use TLS only. - https://phabricator.wikimedia.org/T236017 (10JMeybohm) 05Resolved→03Open [13:38:39] 10serviceops, 10Add-Link, 10Growth-Team: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10kostajh) [13:39:01] 10serviceops, 10Add-Link, 10Growth-Team: Add Link engineering: Allow external traffic to linkrecommendation service - https://phabricator.wikimedia.org/T269581 (10kostajh) [13:39:20] 10serviceops, 10CX-cxserver, 10Language-Team (Language-2020-October-December), 10Patch-For-Review, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10KartikMistry) [16:17:59] Is there a Wikimedia account for Docker to use for Docker image pulls? [16:18:30] My team is running into pull limits under Docker's new scheme but it appears that open source orgs can have an account that has higher limits... [16:18:58] which images are these, aezell_? [16:19:09] Let me get the exact names. [16:19:11] One sec. [16:21:06] I'm not deeply familiar here, but I know in production we pull exclusively from our own self-hosted registry [16:21:17] I'm guessing these are for local dev work? [16:21:33] Ahhh. This is for The Wikipedia Library. It does pulls on its deployments. [16:21:45] The rate limiting means that sometimes those deployments fail. [16:21:57] Let me get some more details from the engineers. [16:22:31] Specifically, it pulls these builds: https://hub.docker.com/r/wikipedialibrary/twlight/tags?page=1&ordering=last_updated [16:23:01] And then, these are deployed to wmflabs/toolforge. [16:23:23] Eventually, to end up at https://wikipedialibrary.wmflabs.org/ [16:24:23] In just reading around, it appears Docker offers a "Open Source Org" type of account with higher rate limits. Mostly, I was just wondering if someone had already created one of these and if we could use it. [16:24:45] I can go through the work of creating one if we decide that makes sense. [16:26:53] serviceops is in a team meeting, if you don't mind holding on for another 20-30m or so :) [16:27:54] Apologies. I'm not in any rush. [16:28:12] no apologies needed! just didn't want you to feel ignored [16:28:37] I'm in bed with a cold just tying up loose ends. Happy to turn the music up and hang out. :) [16:42:13] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 3 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10jijiki) Given that the total size of our redis cluster is now ~2G ( as discussed in T252391#6647730) we can upgrade a few more memc... [16:54:19] * legoktm waves [16:56:10] aezell_: I don't know if WMCS has created a wikimedia account for docker (I 'd say ask them in #wikimedia-cloud), but SRE certainly doesn't have an account for Dockerhub. Production doesn't rely on Dockerhub at all in fact. [16:57:55] That's exactly what I needed to know. Thanks! [16:58:09] you are welcome [16:58:36] Hope y'all are doing well and staying healthy. Happy Holidays! [17:03:02] Happy Holidays to you too! [17:37:59] akosiaris: just to close the loop, WMCS does not have an official Dockerhub account either. We do have some account on quay.io for PAWS things, but nothing for general usage by staff or community. [17:38:27] bd808: I guessed so, but thanks for confirming! [17:38:42] I did not know about the quay.io stuff, TIL [18:10:36] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2035.codfw.wmnet ` The log... [18:25:05] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2035.codfw.wmnet'] ` Of which those **FAILED**: ` ['mc2035.codfw.wmnet'... [18:34:29] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2035.codfw.wmnet ` The log... [18:59:57] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2035.codfw.wmnet'] ` and were **ALL** successful. [19:23:58] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1035.eqiad.wmnet ` The log... [19:48:33] 10serviceops, 10Operations, 10Platform Engineering, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Stretch/Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1035.eqiad.wmnet'] ` and were **ALL** successful. [20:20:27] Icinga alerted about systemd state on mwmaint1002 - that translates to "one of the MW maint timers failed". it was "mediawiki_job_wikidata-updateQueryServiceLag" [20:20:51] but there is also ongoing work on wdqs servers. upgrades to buster. so I think it's due to that and will recover