[08:51:37] !log tools.meetingtimes Updating to version [08:51:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meetingtimes/SAL [09:04:19] !log admin [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev (T251604) [09:04:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:18:21] akosiaris: back then we created a buster-wikimedia thirdparty/kubeadm-k8s repo to hold toolforge k8s stuff. I don't think you use this anywhere in prod, right? We are about to rename the component to something else, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/593499 [09:44:44] !log phabricator - created backups and shut down instance phab-tin (T236571) [09:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [09:44:47] T236571: "phabricator" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236571 [09:45:07] ^ not really jessie, stretch that was upgrade in-place, but anyways [10:25:53] !log phabricator - created backups and shut down instance phabricator (T236571), uploading backups to puppet-phabricator [10:25:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [10:25:56] T236571: "phabricator" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236571 [10:49:38] !log phabricator - uploading backups from previous instances to root on deploy-1002 in the devtools project [10:49:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [11:02:41] !log phabricator - removed instances - replaced by instances in devtools (T236571) [11:02:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [11:02:44] T236571: "phabricator" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236571 [13:02:46] !log toolsbeta `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project [13:02:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:06:32] !log puppet-diffs updated facts from the tools project: `PUPPET_MASTER="tools-puppetmaster-02.eqiad.wmflabs" modules/puppet_compiler/files/compiler-update-facts` [13:06:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/SAL [13:51:28] !log phabricator deleting project and all associated resources on paladox's suggestion [13:51:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [16:05:40] !log tools.cdnjs restarting the service as "canonical" [16:05:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cdnjs/SAL [16:32:47] \o/ [16:33:12] * Lucas_WMDE needs to update his tools for the new cdnjs URL ^^ [16:33:24] (well, maybe s/needs/wants/ ;) ) [16:34:10] hm, actually – what’s the tools-static equivalent under toolforge.org? [16:36:24] Lucas_WMDE: good exam question. `tools-static.wmflabs.org` is currently not affected by the toolforge.org migration [16:36:41] ok, then I guess my tools can stay as they are for now [16:36:42] (since is not running on `tools.wmflabs.org/whatever`) [16:36:56] thanks! [16:43:04] !log tools removing the now unused `/etc/apt/sources.list.d/toolforge-k8s-kubeadmrepo.list` file (T250866) [16:43:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:43:07] T250866: Stage packages for upstream kubeadm v1.16.9 to use in Toolforge - https://phabricator.wikimedia.org/T250866 [16:46:03] !log tools removing the now unused `/etc/apt/preferences.d/toolforge_k8s_kubeadmrepo*` files (T250866) [16:46:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:15:54] Hey, would it be possible for one of the WMCS gods to move an instance from cloudvirt1005 to cloudvirt1012+? integration-agent-puppet-docker-1002.integration.eqiad.wmflabs is the new non-jessie CI runner for puppet but it's on the slow hardware per T223971. :-( [17:15:54] T223971: Old cloudvirt (with Intel Xeon) are half the speed of newer ones (Intel Sky Lake) - https://phabricator.wikimedia.org/T223971 [17:20:27] James_F: seems possible if there is space on one of the less crappy hypervisors. Moving is not instant; it requires shutting down the instance and rsync of its disk to the new hypervisor node. Anything special need to be done before shutting the instance down? [17:21:01] bd808: Nope, I've disconnected it. Thank you! [17:21:15] Don't worry, you won't break SRE's world. ;-) [17:25:01] James_F: that instance is on cloudvirt1015. [17:25:10] https://openstack-browser.toolforge.org/server/integration-agent-puppet-docker-1002.integration.eqiad.wmflabs [17:25:35] bd808: Thank you! [17:25:44] I didn't do anything :) [17:26:07] Oh. [17:26:14] * Reedy giggles [17:26:15] hashar said it was on 1005. [17:26:25] And that we needed to move it… [17:27:03] bd808: Thanks anyway. ;-) [17:27:22] yw ;) [17:29:29] James_F: you can tell hashar that the hardware orders are in to replace cloudvirt100[1-9]. Someday™ the hardware will arrive, get racked, and be pooled. Then he can find a new scapegoat for CI backlog. [17:30:39] * James_F laughs. [17:31:14] FWICS the new image (stretch on 1015) runs puppet jobs about 40% faster than on the old image (jessie on 1005) [17:31:24] Will need to get more data, but things look good. [17:33:41] (Might be jessie vs. stretch or old vs. new docker or sunspots or happenstance; the speed of your instances may go down as well as up; etc.) [21:55:49] !log tools.tool-db-usage Hard restart to create toolforge.org ingress [21:55:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tool-db-usage/SAL [22:08:23] !log tools deleting tools-elastic-01/2/3 T236606 [22:08:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:08:27] T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster - https://phabricator.wikimedia.org/T236606 [22:10:34] !log tools.tool-db-usage Hard restart to set --canonical on [22:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tool-db-usage/SAL [22:36:25] !log tools.mysql-php-session-test Hard restart to create toolforge.org ingress [22:36:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mysql-php-session-test/SAL [22:37:30] !log tools.mysql-php-session-test Hard restart to set --canonical [22:37:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mysql-php-session-test/SAL