[06:53:52] Quick question, Toolforge should have any issues if I connect to a OpenAI endpoint right ? [06:54:04] (Using my own keys) [07:15:09] -- Pod 'generate-new-data-daily-28696730-f8pcv'. Phase: 'pending'. Container state: 'waiting'. With reason 'ErrImagePull'. With message: 'rpc error: code = Unknown desc = failed to pull and unpack image "tools-harbor.wmcloud.org/tool (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)-wsstats/tool-wsstats:latest (http [07:15:09] //tools-harbor.wmcloud.org/tool-wsstats/tool-ws [07:15:10] stats:latest)": failed to resolve reference "tools-harbor.wmcloud.org/tool (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)-wsstats/tool-wsstats:latest (http://tools-harbor.wmcloud.org/tool-wsstats/tool-wsstats:latest)": failed to authorize: failed to fetch anonymous token: unexpected status: 504 Gateway Time-out'. Is this intended ? [07:32:41] dhinus: ^ expected due to outage? [07:33:14] RhinosF1: expected yes, tools-harbor is unresponsive [07:33:30] !status tools-harbor is unresponsive [07:58:15] !status toolforge ongoing outage, builds not working, new pods might fail to start [08:06:06] !status ok [11:51:33] !help It is mentioned on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database [11:51:33] If you don't get a response in 15-30 minutes, please email the cloud@ mailing list -- https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication [11:51:33] that "Database credentials are generated on account creation and placed in a file called replica.my.cnf in the home directory of both a Tool and a Tools user account. This file cannot be modified or removed by users." [11:51:34] I have done ssh into toolforge and also done "become mytool" [11:51:34] But where can i see my replica.my.cnf file. [11:51:35] I am really struggling with https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases. [11:52:29] Guest61: what is your tool name? [11:53:01] I have just created a tool wikievolution. [11:53:08] Not added any code so far. [11:53:23] I want to create a database for my tool on tools db [11:57:03] let me check a few things [11:59:14] !log admin restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing [11:59:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:59:22] Guest61: you should be all set now [11:59:43] Guest61: you have a replica.my.cnf file now in your tool home dir [12:00:23] Yeah thankyou :) [12:01:02] np [12:04:56] when trying to connect to tools.db.svc.wikimedia.cloud with the replica.my.cnf credentials, [12:04:56] it says: [12:04:57] ERROR 1045 (28000): Access denied for user 's56037'@'172.16.1.16' (using password: YES) [12:21:02] I mean after doing "beome mytool" I am not able to connect to tools.db, but by directly connecting to tools.db through my toolforge account, i am able to connect. Why is it so? [12:21:56] yep, it did not work [12:21:58] Jul 24 12:20:57 cloudcontrol1005 maintain-dbusers[1353413]: ERROR [root.populate_accountsdb:756] problem populating new account: Request to create replica.my.cnf file for account_type tool and account_id tools.wikievolution failed without response. [12:22:00] looking [12:27:08] Guest61: done :), can you try now? [12:27:35] the fix is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056489, though I want to see why it was not caught by the functional tests before merging [12:28:26] hmm... the tests don't pass :/ [12:28:36] Yeah i am able to do it now [12:29:03] will investigate, Guest61 thanks for your patience! let us know if there's anything else that does not work :) [12:29:27] Yeah Thankyou so much for your quick response.:) [16:02:45] !log deployment-prep moved sessionstorage/kask from sessionstorage04 to sessionstorage06 T370461 [16:02:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [16:02:49] T370461: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461 [19:12:53] I don't understand why the bookworm instances I created on 2024-07-02 are already being displayed as "deprecated 2024-07-03" in horizon? [19:20:50] don-vip: it just means that a new version of the bookworm image has been published since you created your instances [19:21:59] occasionally new image versions are created to apply updates etc so newly-created instances have those updates preinstalled - there shouldn't be any need to worry about it [19:32:17] ok thank you, it worried me :) [19:56:45] !log video Shutting down all celery workers to migrate the Redis database, then encoder instances [19:56:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:09:17] I have a cloud VPS project with a local mariadb install. The setup is also puppetized, don't have a particular issue with it. Except I should have offsite backups and not having to maintain mariadb wouldn't hurt. Would you say I should still look at using trove instead? [20:28:54] mutante: trove will not give you backups as far as I know [20:37:09] bd808: I just created a trove instance out of curiosity and saw a "Create Backup" button next to it.. just now wondering [20:37:46] I am debating if it's making things simpler or more complex to use the extra instance vs just keeping local mariadb ... [20:38:15] https://wikitech.wikimedia.org/wiki/Help:Trove_database_user_guide#Backup_or_snapshot_your_databases [20:38:25] ah, thanks [20:39:15] the task that the snapshot line mentions is resolved so ¯\_(ツ)_/¯ [20:39:21] I do get a "Backup Database" pop-up window when hitting that button [20:39:39] asks for swift container name [20:40:35] !log video drop gfg and encoding07 instances (unused) as per T360711 + T365154 [20:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:40:39] T360711: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711 [20:40:40] T365154: video2commons general failure - https://phabricator.wikimedia.org/T365154 [20:40:58] *nod* seems like that task was to enable this but we would still need swift [20:41:24] !log video migrated redis database from video-redis-buster to video-redis-bookworm as per T360711 + T365154 [20:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [20:41:33] I should probably just get that cinder volume and store my mysqldump there [21:42:45] I am trying to let puppet mount a cinder volume. I copied the code from profile::labs::cindermount::srv and just adjusted the mountpoint. The command it runs fails with "Refusing to format volume in non-interactive mode without --allow-unattended-format" [21:44:18] sounds like that option has to be added somewhere. though .. when I run that myself it feels like that is "interactive" [21:45:02] puppet just runs the /usr/local/sbin/wmcs-prepare-cinder-volume here [21:45:23] I need a disk quota incrase to complte the video bookworm migration ^^' https://phabricator.wikimedia.org/T370964 [22:37:14] don-vip, just the storage quota change or do you also need more VM things? [22:42:35] !log devtools gitlab-prod-1002 out of disk again when attempting package upgrade - apt-get clean; rm /var/log/syslog.2*.gz rm /var/log/messages.2*.gz to get space [22:42:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [23:38:59] andrew sorry I didn't see your question here. For now only disk, thank you :) [23:39:45] !log video updated and restarted video2commons frontend to use the bookworm redis instance. Setup encoding01 as the first new bookworm instance as per T360711 + T365154 [23:39:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:39:49] T360711: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711 [23:39:50] T365154: video2commons general failure - https://phabricator.wikimedia.org/T365154 [23:40:15] I hope video2commons is still working... [23:40:33] I'lee see the outcome of my actions tomorrow, it's late here. Good night :)