[03:20:19] !log tools.wsexport Updated to 2d9d41083d33714afff99dfcddcb0a739570600a [03:20:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wsexport/SAL [04:44:03] !log wmf-apt Replaced mwv-apt-01.mwv-apt.eqiad.wmflabs with mwv-apt-02.mwv-apt.eqiad.wmflabs (T236538) [04:44:05] bd808: Unknown project "wmf-apt" [04:44:05] T236538: "mwv-apt" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236538 [04:44:19] !log mwv-apt Replaced mwv-apt-01.mwv-apt.eqiad.wmflabs with mwv-apt-02.mwv-apt.eqiad.wmflabs (T236538) [04:44:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwv-apt/SAL [04:56:21] !log grantreview Built grantreview-04 (T236542) [04:56:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Grantreview/SAL [04:56:26] T236542: "grantreview" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236542 [11:16:16] !log tools.wd-image-positions deployed 520c618b5a (work around T222159); temporary experiment still in effect [11:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [13:24:07] Hi cloud-team - I'm having an issue connecting to newly launched instance (spagnite1-1.analytics.eqiad.wmflabs) - It tells me "Permission denied (publickey)" [13:24:17] Any idea? [13:37:48] Hi joal how long after creating the instance did you try? The first boot configuration process can take a few minutes [13:38:40] Hi jeh - Waited some - instances (spagnite1-1 and spagnite1-2) have started like 40 mins and 25mins ago [13:39:26] any problems logging into `spagnite1-2`? [13:40:15] jeh: same as 1-1 [13:41:23] I see these systems had a puppet catalog error on boot (can be seen from the console log in Horizon) [13:41:26] `Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Cluster analytics_eqiad_wmflabs not defined in wikimedia_clusters at /etc/puppet/modules/profile/manifests/base.pp:51:9 on node spagnite1-2.analytics.eqiad.wmflabs` [13:42:30] hm [13:43:23] I guess this is a puppet error in analytics side, right? [13:44:54] it looks that way. Due to that, the puppet agent isn't able to complete the configuration which includes ssh access [13:55:56] makes sense jeh [13:56:14] I'll drop the failed instances and check with the team how comes this error happens [13:56:20] Thanks a lot! [13:59:38] no problem, let us know if we can help out [14:14:02] joal: looks like the wikimedia_cluster hiera key recently changed on labs, https://phabricator.wikimedia.org/T234232 [14:15:30] I can't say for sure, but you may want to remove the `cluster: analytics_eqiad_wmflabs` hiera key from the analytics cloudvps project in horizon. [14:15:35] jeh: right - But the issue is set to resolved :( [14:15:49] hm [14:15:49] that was resolved for the `misc` project [14:16:00] sorry, `misc` cluster [14:16:07] jeh: I'm not good enough in how we have puppet setup to take that decision :) [14:16:32] jeh: I'll wait for my beloved ops in order not to break everything [14:19:14] thanks again jeh for looking into this [14:19:45] you're welcome, happy to help [14:27:58] !log openstack failover cloudnet2002-dev's neutron router to cloudnet2003-dev - testing T214297 [14:28:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:28:02] T214297: cloudvps: neutron: investigate keepalived warning - https://phabricator.wikimedia.org/T214297 [18:22:34] I am leading a session at Tech Conf on 2019-11-13 about "On-wiki documentation & documentation as code: strengths, weaknesses, and compromises". Folks are welcome to read the task (T234634) and provide their thoughts related to the 3 main questions posed whether they will be at the conference in Atlanta or not. [18:22:35] T234634: Wikimedia Technical Conference 2019 Session: On-wiki documentation & documentation as code: strengths, weaknesses, and compromises - https://phabricator.wikimedia.org/T234634 [19:11:20] Hi All, sorry to have forgotten this, but how do I request a floating (or static) IP for my VPS instance? Thanks [19:21:58] Hi notconfusing we have some information on that at https://wikitech.wikimedia.org/wiki/Help:Addresses [20:38:40] !log tools Updated A record for tools-static.wmflabs.org to point towards project-proxy T236952 [20:38:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:38:45] T236952: Move tools-static.wmflabs.org behind project-proxy - https://phabricator.wikimedia.org/T236952 [20:56:25] !log tools Removed tools-trusty.wmflabs.org CNAME to login-trusty.tools.wmflabs.org as that target record does not exist, presumably deleted ages ago [20:56:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:57:48] !log tools Removed trusty.tools.wmflabs.org CNAME to login-trusty.tools.wmflabs.org as that target record does not exist, presumably deleted ages ago [20:57:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:00:45] !log tools Removed tools-checker.wmflabs.org A record to 208.80.155.229 as that target IP is in the old pre-neutron range that is no longer routed [21:00:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:08:48] !log wmflabsdotorg Removed old internal-server-nat.wmflabs.org. A record to 208.80.155.255 - the target IP is in the old pre-neutron range that is no longer routed, this name has been replaced in T234836 [21:09:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmflabsdotorg/SAL [21:09:01] T234836: CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress - https://phabricator.wikimedia.org/T234836 [21:15:06] !log paws Updated paws-apiserver.wmflabs.org A record list to remove 172.16.2.151 which is not allocated to any instance. The other two A records point to valid instances in the paws project. [21:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [23:36:19] Is python 3.6 or better not available at all, or do I need to activate it somehow? [23:45:19] Oh I guess not at all, I see T230961 is a request to get it [23:45:20] T230961: Install a version of Python newer than 3.5.3 in Toolforge - https://phabricator.wikimedia.org/T230961