[00:05:29] !log admin Running wmcs-novastats-dnsleaks (T262359) [00:05:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [00:05:35] T262359: tools-redis-1004.tools.eqiad1.wikimedia.cloud resolves to multiple IPs - https://phabricator.wikimedia.org/T262359 [04:38:30] Hi. This might can help in US Election. https://ibb.co/9rFTNYr [10:38:56] !log tools created server group `tools-ingress` with soft anti affinity policy (T250172) [10:39:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:39:02] T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes - https://phabricator.wikimedia.org/T250172 [10:42:01] !log tools created VMs tools-k8s-ingress-1 and tools-k8s-ingress-2 in the `tools-ingress` server group T250172) [10:42:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:50:26] !log tools created puppet prefix `tools-k8s-ingress` (T250172) [10:50:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:50:32] T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes - https://phabricator.wikimedia.org/T250172 [11:12:21] !log tools new ingress nodes added to the cluster, and tainted/labeled per the docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Deploying#ingress_nodes (T250172) [11:12:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:12:27] T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes - https://phabricator.wikimedia.org/T250172 [11:24:37] !log toolsbeta created new puppet prefix `toolsbeta-test-k8s-ingress` (T250172) [11:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:24:42] T250172: Toolforge: k8s: ingress: consider creating ingress-specific nodes - https://phabricator.wikimedia.org/T250172 [11:25:50] !log toolsbeta created new server group toolsbeta-k8s-ingress (T250172) [11:25:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:27:56] !log toolsbeta created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 (T250172) [11:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:39:23] !log toolsbeta all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them [11:39:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:42:06] !log toolsbeta actually, the whole k8s cluster seems down? the API seems down at least [11:42:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:45:09] !log toolsbeta force-rebooting the 3 k8s etcd nodes. They seem down [11:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:50:42] !log toolsbeta after force-rebooting everything, the k8s cluster seems to have recovered itself. magic. [11:50:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:10:54] Hello, I am still new with Toolforge. I have some issue linking my Django to the Database. So I have a database "maria_cyclingdb", let's say my user name is s12345. In Django, I filled it that way DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'maria_cyclingdb', 'USER': DB_USER, [13:10:54] 'PASSWORD': DB_SECRET_KEY, 'HOST': 'tools.db.svc.eqiad.wmflabs', 'PORT': '', } } [13:11:12] and I get following error: django.db.utils.OperationalError: (1045, "Access denied for user 's54511'@'172.16.7.167' (using password: NO)") [13:11:17] when I migrate [13:12:04] I can connect to my database using "sql tools" [13:12:23] mysqlclient is installed [13:13:23] (using password: NO) [13:13:38] but DB_SECRET_KEY is not empty [13:13:39] Sounds like it's not passing a password [13:14:39] is the name of the db: "maria..." or "s12345__maria..." [13:14:44] for Django? [13:14:56] you need to use the full name (so including the username prefix) [13:14:56] how can I test that the password is passed? [13:15:01] ok [13:16:20] django.db.utils.OperationalError: (1045, "Access denied for user 's54511'@'172.16.7.167' (using password: YES)") [13:16:26] some progres... [13:18:57] And a mistake in the password [13:19:03] ok it works thanks! [14:25:24] paladox, puppet is broken on phabricator-stage-1001.devtools.eqiad1.wikimedia.cloud and phabricator-prod-1001.devtools.eqiad1.wikimedia.cloud — those are yours, right? [14:26:03] if so can you see about resolving the problems? [14:26:18] oh, yeh [14:27:14] thanks! [14:27:16] Do you know what the error says? [14:27:41] (since i have a new mac so have to setup ssh for it again + new key needs generated) [14:27:45] it's about a missing user [14:27:51] oh [14:28:00] not urgent, I'm just doing some other cleanup and noticed [14:28:11] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'User[deploy-devtools]' in parameter 'require' (file: /etc/puppet/modules/scap/manifests/target.pp, line: 188) on node phabricator-stage-1001.devtools.eqiad.wmflabs [14:36:29] andrewbogott i think https://gerrit.wikimedia.org/r/c/operations/puppet/+/626035 may fix that [17:40:23] !log admin giving ceph pg autoscale another chance: ceph osd pool set eqiad1-compute pg_autoscale_mode on [17:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:01:05] !log restarting ceph-mgr@cloudcephmon1003 in hopes that the slow ops reported are phantoms (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/) [18:01:05] andrewbogott: Unknown project "restarting" [18:01:15] !log admin restarting ceph-mgr@cloudcephmon1003 in hopes that the slow ops reported are phantoms (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/) [18:01:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:13:36] !log admin restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms [18:13:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:14:13] !log transferpy-test deleting and cleaning up resource per request #T262418 [21:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Transferpy-test/SAL [21:14:19] T262418: Request deletion of transferpy-test VPS project - https://phabricator.wikimedia.org/T262418 [21:14:55] * balloons notes the stashbot comments in phab are really nice