[07:27:11] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10Joe) @elukey we can do the same for etcd, as long as we don't use codfw for clients. So my proposal would be to: - change dns/configuration of every etcd client to point to eqiad - reimage one server at a tim... [07:27:17] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10Joe) p:05Triage→03High [07:35:17] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10elukey) Seems good! For Zookeeper it should be a matter of reimaging one node at the time, conf200x are already running the stretch version backported, so there shouldn't be any upgrade to do in theory. Data f... [07:44:10] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10elukey) Also in the procedure we should remember to stop and downtime `etcdmirror` on conf2002 when we reimage the node. [08:03:34] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1030.eqiad.wmnet ` The log can be found i... [08:03:50] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2030.codfw.wmnet ` The log can be found i... [08:16:28] 10serviceops, 10SRE, 10docker-pkg: Docker image on the build host seem to ignore apt priority for wikimedia packages - https://phabricator.wikimedia.org/T268612 (10Joe) 05Open→03Resolved [08:41:35] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2030.codfw.wmnet'] ` and were **ALL** successful. [09:33:35] 10serviceops, 10SRE, 10docker-pkg: docker-pkg update cli renders unclear guidance - https://phabricator.wikimedia.org/T253131 (10Joe) a:03Joe I agree @rzl, your proposal is probably the most user-friendly solution. Rationale here was I felt I had no need to replicate in every subparser the same last argume... [10:46:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubestage200 - https://phabricator.wikimedia.org/T271702 (10JMeybohm) [10:47:44] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubestage200* change on every puppet run - https://phabricator.wikimedia.org/T271702 (10JMeybohm) a:05akosiaris→03None [11:01:05] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubestage200* change on every puppet run - https://phabricator.wikimedia.org/T271702 (10jbond) The following is the line that iptables-save sees which is added by cali and not avalible in ferm. we need to filter this out of ferm-status some how ` -A FORWARD -m... [11:02:17] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: kubestage200* change on every puppet run - https://phabricator.wikimedia.org/T271702 (10jbond) [12:08:40] should create_new_service.sh in deployment-charts be using the 0.2 scaffold these days? [12:49:11] hnowlan: ah, sorry for not answering wherever you asked that last week :) - it def. should! [12:49:35] It should always use the latest version I guess [14:53:44] <_joe_> hnowlan: doesn' [14:53:46] <_joe_> t it? [14:53:53] <_joe_> I missed the discussion here [15:13:28] 10serviceops, 10MW-on-K8s, 10SRE, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Pipeline): Deployment infrastructure for PHP microservices - https://phabricator.wikimedia.org/T261369 (10akosiaris) >>! In T261369#6707040, @Legoktm wrote: >>>! In T261369#6695002, @akosiaris wrote: >>>>! In T261... [15:15:03] hnowlan: yes. Patch is up at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/643057 never got around to merge it though [15:39:57] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc1028.eqiad.wmnet ` The log can be found i... [15:40:10] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` mc2037.codfw.wmnet ` The log can be found i... [16:06:34] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc1028.eqiad.wmnet'] ` and were **ALL** successful. [16:19:35] 10serviceops, 10Platform Engineering, 10SRE, 10Wikidata, and 4 others: Upgrade memcached cluster to Debian Buster - https://phabricator.wikimedia.org/T213089 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mc2037.codfw.wmnet'] ` and were **ALL** successful. [17:57:42] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Dzahn) @Reedy "stretch -> buster" and "PHP 7.2 -> PHP 7.3" are separated deliberately. So yes, they should also be a... [18:01:41] 10serviceops: Migrate WMF Production from PHP 7.2 - https://phabricator.wikimedia.org/T271736 (10Reedy) [18:01:55] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Reedy) [18:02:08] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Reedy) [18:02:14] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Reedy) [18:02:27] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Reedy) [18:02:30] 10serviceops, 10Release-Engineering-Team, 10PHP 7.2 support, 10Patch-For-Review: Drop PHP 7.2 support from MediaWiki master branch, once Wikimedia production is on 7.3 - https://phabricator.wikimedia.org/T261872 (10Reedy) [18:02:52] 10serviceops, 10Release-Engineering-Team, 10PHP 7.2 support, 10Patch-For-Review: Drop PHP 7.2 support from MediaWiki master branch, once Wikimedia production is on 7.3 - https://phabricator.wikimedia.org/T261872 (10Reedy) [18:02:56] 10serviceops, 10Release-Engineering-Team-TODO, 10SRE, 10Patch-For-Review, and 2 others: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Reedy) [18:04:46] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Reedy) 05Open→03Stalled [18:04:50] 10serviceops, 10Release-Engineering-Team, 10PHP 7.2 support, 10Patch-For-Review: Drop PHP 7.2 support from MediaWiki master branch, once Wikimedia production is on 7.3 - https://phabricator.wikimedia.org/T261872 (10Reedy) [18:05:10] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Reedy) Marking stalled as we're a bit away from this happening as {T245757} isn't finished yet [20:46:59] when updating a docker image, and there is a version like: 2.4.38-1-s2 when would I change the "1" and when would I change just the "s2" [22:53:21] mutante: 2.4.38 = version of software, 1 is version of the image's config, s2 is number of image rebuilds to pick up changes in parent images [23:31:17] legoktm: 'k, ACK. then I need to change it again [23:48:03] legoktm: rebuilt the production images one more time. looks like it is fine. thanks [23:48:21] :D [23:49:24] out for now, cya [23:49:30] o/