[06:45:34] <[1997kB]> !help Hi when I try to connect to database of my too via MySQL Workbench, I get "Access denied for user 'u20998'@'%' to database 's53950'" [06:45:34] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [09:25:31] [1997kB]: I think you can only connect to your database from within toolforge [10:15:30] Is logrotate available on tools? [10:15:48] $ logrotate [10:15:48] -bash: logrotate: command not found [10:18:56] hauskatze: no [10:19:11] for your own tool log files? [10:19:20] arturo: yep [10:19:39] I `truncate` them manually after some time [10:21:04] I could envision several improvements related to how tool developers interact with their software, and I'm sure others in my team and in the community too. Exiting times ahead :-) [10:21:11] hauskatze: I take you are using the grid, right? [10:21:32] arturo: to submit jobs? sure [10:21:57] I mean, kubernetes-based tools may benefit from `kubectl logs` [10:22:12] I think tools.wikibugs do have something [10:28:35] I took a look a while ago to migrate my cron jobs from crontab to kubernetes and it was uh oh, too complex [11:29:28] yeah, I don't think cron jobs are fully supported in our particular k8s deployment yet [13:27:51] <[1997kB]> arturo: I am following https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_with these instructions. [13:34:21] [1997kB]: will take a look later, perhaps your credentials are messed up [14:25:15] [1997kB]: You are trying to use your user's credentials to access a schema that is owned by another user and not marked as public (ending in __p). That's the problem. [14:26:21] and 's53950' would not actually be possible to create in the first place. It would have to be 's53950__' [14:28:10] <[1997kB]> yeah, That was s53950__SWViewer [14:30:18] <[1997kB]> so I need to ask creator of schema to make it public? [14:30:48] if you want to access it from another user, yes [14:52:05] hmm, it appears the letsencrypt script is broken [14:52:08] getting: Notice: /Stage[main]/Profile::Gerrit::Server/Letsencrypt::Cert::Integrated[gerrit]/Exec[acme-setup-acme-gerrit]/returns: "detail": "Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.", [14:54:05] o/ is it possible to add an ldap group as a tool maintainer? :D [14:56:35] addshore: this sounds like an XY problem question. What is your goal? [15:08:57] !log admin trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589 [15:09:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:11:03] is there some command like jsub to kubernetes to run script in a simple way without have to create a yaml file? [15:11:53] danilo: no, we have not built any fancy wrappers for that yet. [15:12:50] is there some task about that? can I create one? [15:13:43] kubernetes is too complicated, a command like taht would help a lot [15:15:46] danilo: we have some tasks like T249787 [15:15:47] T249787: Create Docker image for Toolforge that is purpose built to run pywikibot scripts - https://phabricator.wikimedia.org/T249787 [15:16:12] its is a known gap, but also not something that is going to happen really soon [15:20:43] bd808: to be able to easily add all WMDE people to a tool, (there is an ldap group with them all in) rather than add them all individually [15:21:15] especially considering there are multiple tools to add them all to. I see I can add a tool as a maintainer for another tool, so perhaps if using ldap groups isnt possible that would be the way to go [15:22:04] addshore: ah. there is not a way to do that exactly today, no. A tool can be a maintainer of another tool, so you could consolidate that way, but not with the existing wmde group [15:22:20] cool, thanks! [15:34:27] andrewbogott hi, i'm wondering if you could assign a floating ip to the devtools project please? :) [15:38:45] paladox: according to https://openstack-browser.toolforge.org/project/devtools you have 10 floating IPs quota [15:38:59] (which seems too much BTW) [15:39:10] hmm, 10? [15:39:20] horizion shows me none available [15:39:50] let me check in the CLI [15:40:19] the CLI shows 0 quota, which makes more sense [15:40:47] o/ [15:41:03] https://phabricator.wikimedia.org/F31855122 [15:41:09] arturo ^ [15:41:52] paladox: I think you need to follow the usual workflow for quota reviews in order to request a new floating IP [15:41:58] let me check the docs to see if this is true [15:42:01] should striker.wmflabs.org still be around? I accidentally was trying to use it today? [15:42:39] toolsadmin.wikimedia.org is presumably the recommended place now? [15:43:24] tarrow: yes, toolsadmin. I'm unsure about the striker one, may be either for devel or just an alias. If you use toolsadmin you are in the safer side anyway [15:44:08] paladox: yes, please follow https://phabricator.wikimedia.org/project/view/2880/ [15:46:30] ah, thanks [15:46:59] Done v [15:47:00] * https://phabricator.wikimedia.org/T254596 [15:47:01] arturo: I guess it is just for devel. I was repeatedly trying to log in (like a silly confused person)? Any merit it putting warning banner on it? [15:47:23] tarrow: that's fair. Please open a phab task requesting it [15:47:39] Roger, any specific project you want it under? [15:48:22] tarrow: striker is my dev/qa project for toolsadmin [15:48:50] paladox: noted, thanks. Will get back to you on Wednesday next week after our team meting [15:49:04] ok [15:50:06] arturo: https://phabricator.wikimedia.org/T254598 [15:50:51] bd808: gotcha; I have no idea what led me there (I typed the URL manually from my brain) so it's entirely my fault. No idea if other people also make this mistake [15:51:27] obviously feel free to decline the task :). Just though I might save someone else thinking they had gone mad for 5mins [16:20:39] all irc bots have just cycled the network, did something stutter? [16:23:04] actually, all except one instance of wm-bot (wm-bot2) [16:23:20] mmm [16:23:34] let me check real quick [16:27:20] looks like stashbot_ maybe having trouble talking to LDAP? Kind of unclear. I'm going to restart it and see what happens [16:27:34] yeah, I see the same [16:28:08] however, same LDAP errors yesterday [16:28:42] ldap burps happen, but we did have a nice long period of it being pretty stable [16:29:02] ldap3.core.exceptions.LDAPSessionTerminatedByServerError: session terminated by server [16:30:04] !log tools.stashbot Hard restart for LDAP session termination issues [16:30:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [16:30:23] arturo: any known reason why other bots died? [16:30:36] * RhinosF1 notices they all came back quick [16:30:51] RhinosF1: the logs for wikibugs is clean for example [16:31:40] arturo: yeah, that error could really be any network interruption. ks8 -> cloud internal -> core network -> ganetti on that path [16:31:59] * arturo checks several grafana dashboards [16:32:21] ok, we had a uncontrolled failover from cloudnet1004 to cloudnet1003, i.e, neutron [16:32:25] https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad1?orgId=1&refresh=1m [16:32:55] that would blip everything for sure :) [17:17:01] arturo: wmbot and the zppixbot's both went same tine [17:17:16] That'll be it [17:17:49] Anything knocks the network and it dies [17:54:21] !log tools.stewardbots Restart stewardbot IRC bot [17:54:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [17:59:42] !log tools.stewardbots Kill stewardbot job, so I can debug from commandline [17:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:01:42] !log tools.stewardbots jstart -N stewardbot -mem 2G /data/project/stewardbots/venv-py3/bin/python3 -u /data/project/stewardbots/stewardbots/StewardBot/StewardBot.py [18:01:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:14:57] I now have nothing running on the grid for any of my bots or tools [18:15:10] * AntiComposite is glad to ditch python 3.5 [18:38:37] bd808: I created a new task for a jsub like command for kubernetes with some ideas: https://phabricator.wikimedia.org/T254617 [21:27:24] !log tools.zppixbot kubectl delete pods --all [21:27:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:46:55] was anyone aware that anyone with toolforge access can read all files? even if you are not actually on the list of maintainers for the tool? [21:47:14] Texas: yes [21:47:14] arturo, bd808: ^ [21:47:19] ah [21:48:00] you can change filesystem permissions (and should for secrets) but things are intended to be as open as possible [21:48:11] ah okay [21:48:29] so I can use chmod to disable read access to a config file with passwords? [21:48:34] yes [21:48:37] and pelase do! [21:53:19] thank you so much bd808 [21:53:32] I was very unsettelled when I discovered this [21:53:40] !log tools.zppixbot chmod a-r default.cfg [21:53:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:56:14] !log tools.zppixbot chmod o-r default.cfg [21:56:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [21:58:23] !log tools.zppixbot-test restarting for code and config changes [21:58:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [22:18:25] !log tools.zppixbot restarting for code and config changes [22:18:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [22:28:39] !log tools.zppixbot remove junky reminder entries [22:28:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [22:43:58] Texas: when you remove reminders you have to restart, otherwise it will still say the reminder [22:44:58] oh [22:45:17] I believe the guide says that as well [22:45:32] Or, it should, since I wrote it [22:45:53] !log tools.zppixbot kubectl delete pods --all [22:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL