[10:10:56] !log admin Restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading (T274565) [10:11:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:11:01] T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565 [10:29:48] !log admin Done restoring the 4.9 kernel on cloudcephosd2001-dev and upgrading, requires logging into console to boot from the older kernel before removing the newer one (T274565) [10:29:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:29:52] T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565 [12:02:53] Hi, it seems that #wikimedia-labs is invite-only, yet I've been told at my userpage to go there for any help... [12:03:19] Examples: https://wikitech.wikimedia.org/wiki/User_talk:Tim_Landscheidt and https://wikitech.wikimedia.org/wiki/User_talk:Leaderboard [12:05:42] My question is whether we can access a GUI desktop using bastion - I know I can use ssh, but was wondering on the lines of RDP [12:07:10] Leaderboard: those are outdated messages from several years ago, #wikimedia-labs is now #wikimedia-cloud [12:07:51] I don’t know why it’s customary to substitute “welcome” templates, it’s bound to produce problems like this [12:08:02] (or outdated project logos – tons of user pages still link to the Wikipedia v1 logo) [12:09:13] Is the 2 GB RAM/16 GB HDD limit for everyone or was that an artifact on when I requested access (in 2013)? [12:09:47] I think that’s outdated by now [12:10:23] Lucas_WMDE: do you mean that users are supposed to have more RAM/SSD now (than what I've got)? [12:11:01] Leaderboard: on toolforge or on a cloud vps virtual machine? [12:11:13] Majavah: on bastion.wmflabs.org [12:11:36] you shouldn't really be doing anything on the bastion except to proxy to other vms [12:12:25] yeah [12:15:19] !log admin Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading (T274565) [12:15:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:15:24] T274565: [ceph] Test and upgrade to kernel ~15 - https://phabricator.wikimedia.org/T274565 [12:18:00] Leaderboard: looks like your account doesn't have access to anything else than the bastion hosts, that shouldn't even be possible for new accounts today - what do you want to run on cloud? that would help when trying to explain where to run things / apply [14:54:20] !log toolsbeta shrinking etcd cluster to 3 members, cleaning up automation runs [14:54:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:17:27] !log toolsbeta etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook) [15:17:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:18:57] !log tools adding new etcd member using the cookbook wmcs.toolforge.add_etcd_node (T267082) [15:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:19:01] T267082: Rebuild Toolforge servers that should not have NFS mounted (and with affinity) - https://phabricator.wikimedia.org/T267082 [15:19:43] * bstorm never got around to messing with etcd yesterday at all [15:20:04] I found a couple bugs :), you have patches for review now xd [15:21:06] Cool :) [15:21:09] Finding bugs is good [15:22:10] the security groups for etcd on tools project are a bit mixed :/ [15:22:57] (btw. it takes rather long to add/remove a node, as every time it has to refresh the ssl certs of all the other nodes, and that means all the puppet dance) [15:30:07] yeah, it needs to resync the cluster anyway, so taking a while is probably a good thing. If it seemed too fast, you might be tempted to collapse the thing, right? 😁 [15:35:58] yep, I'm still working on taking the health of the cluster into account, do you know if there's a way to see if the cluster is moving stuff or if it's already stable? (will the 'healthy'-'unhealthy' simplicifation show it?) [15:36:26] !log tools Added new etcd member tools-k8s-etcd-9.tools.eqiad1.wikimedia.cloud (T267082) [15:36:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:36:31] T267082: Rebuild Toolforge servers that should not have NFS mounted (and with affinity) - https://phabricator.wikimedia.org/T267082 [15:38:59] oh, I might have added it with nfs too... xd [15:40:20] hiera says 'mount_nfs: false' looking on the node... [15:40:38] awesome, no nfs :) [15:43:29] !log tools Removing etcd member tools-k8s-etcd-5.tools.eqiad.wmflabs (T267082) [15:43:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:43:34] T267082: Rebuild Toolforge servers that should not have NFS mounted (and with affinity) - https://phabricator.wikimedia.org/T267082 [15:46:38] :) [15:53:31] !log tools Removed etcd member tools-k8s-etcd-5.tools.eqiad.wmflabs, adding a new member (T267082) [15:53:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:53:35] T267082: Rebuild Toolforge servers that should not have NFS mounted (and with affinity) - https://phabricator.wikimedia.org/T267082 [18:04:53] !log tools.lexeme-forms deployed f2b128273d (l10n updates) [18:04:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [20:43:35] !log tools cleared error state from the grid queues caused by unspecified job errors [20:43:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL