[00:35:17] <bstorm>	 On s7 only
[00:35:25] <bstorm>	 and heartbeat_p of course :)
[00:35:30] <bstorm>	 You'd have to filter that stuff
[00:36:00] <bstorm>	 !log toolsbeta updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing
[00:36:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[03:56:40] <tanny411>	 9:56 AM <tanny411> Cannot connect to commonswiki. Anything I am missing? `(2003, "Can't connect to MySQL server on 'commonswiki.analytics.db.svc.eqiad.wmflabs' ([Errno 113] No route to host)")`
[03:57:08] <AntiComposite>	 where are you trying to connect from
[04:25:41] <tanny411>	 AntiComposite: From Toolforge
[09:12:15] <JJMC89[m]>	 I'm getting errors about expired certificates for Cloud, including Toolforge.
[09:16:26] <dcaro>	 Hi JJMC89[m], can you give one specific url I can look at?
[09:16:41] <JJMC89[m]>	 dcaro: https://accounts.wmflabs.org/
[09:17:05] <dcaro>	 thanks! looking
[09:19:08] <dcaro>	 created T273956
[09:19:09] <stashbot>	 T273956: Expired certificates in cloud urls - https://phabricator.wikimedia.org/T273956
[09:19:29] <dcaro>	 !log admin Some certs around the infra are expired (T273956)
[09:19:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:19:35] <arturo>	 JJMC89[m]: including toolforge? like `whatever.toolforge.org`?
[10:21:28] <dcaro>	 !log admin This was affecting maps and several others, maps and project-proxy have been fixed (T273956)
[10:21:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:21:31] <stashbot>	 T273956: Expired certificates in cloud urls - https://phabricator.wikimedia.org/T273956
[10:22:17] <dcaro>	 JJMC89[m]: accounts has been fixed too, let me know if you see any others
[10:29:57] <Lucas_WMDE>	 chat.wmcloud.org was also expired but seems to be fixed now
[10:30:19] <arturo>	 yes Lucas_WMDE the shared front proxy certs expired, but that's fixed now
[10:30:40] <arturo>	 it affected every <whatever>.wmcloud.org or <whatever>.wmflabs.org web service
[10:30:47] <Lucas_WMDE>	 ah ok
[10:32:05] <dcaro>	 don't hesitate to ping if you see any more issues though
[10:59:02] <arturo>	 !log admin icinga-downtime labstore1004 tools share space check for 1 week (T272247)
[10:59:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:59:07] <stashbot>	 T272247: 2021-01-17: tools NFS share cleanup - https://phabricator.wikimedia.org/T272247
[13:08:12] <arturo>	 it seems I cannot create gerrit repos
[13:08:46] <arturo>	 I got `create project not permitted` when running the command listed in https://www.mediawiki.org/wiki/Git/Creating_new_repositories#Step_3:_Create_the_repository
[13:09:54] <arturo>	 dcaro: how did you create the ansible repo?
[13:15:16] <arturo>	 anyway, will try next week
[13:25:55] <dcaro>	 arturo: I asked someone iirc, creating a task
[13:28:30] <arturo>	 ack
[15:49:39] <bd808>	 arturo: https://www.mediawiki.org/wiki/Gerrit/New_repositories is the easy way to ask for a new gerrit repo. q.chris is generally pretty quick to respond
[16:36:00] <JJMC89[m]>	 thanks dcaro. arturo: yes, but looks OK now. Toolforge ones were showing something on the screen but it looked like asses were missing and the error was in the browser console.
[16:38:00] <JJMC89[m]>	 *assets
[17:09:55] <wm-bot>	 !log tools.meetbot <bd808> Restarted bot to allow truncation of 532G log file (T273961)
[17:09:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meetbot/SAL
[17:12:30] <Lucas_WMDE>	 wowee
[17:12:40] <bd808>	 I know right?
[17:17:10] <Lucas_WMDE>	 I also pinged the wikidata-analysis people about that other 64GB file in d.caro’s phab comment
[17:17:19] <Lucas_WMDE>	 bc why not
[17:27:34] <Majavah>	 how does a tool even manage to take up half a terabyte of logs
[17:29:31] <Reedy>	 by logging a lot
[17:29:36] <Reedy>	 and not rotating
[17:33:06] <Reedy>	 like owls, you should rotate your logs
[18:06:36] <wm-bot>	 !log tools.meetbot <bd808> Restarted bot to allow truncation of another 532G log file (T273961)
[18:06:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meetbot/SAL
[19:27:09] <Majavah>	 I'm getting 500's on https://k8s-status.toolforge.org/
[19:31:27] <phamhi>	 Majavah: taking a look
[19:37:04] <phamhi>	 It's a bit slow but coming up for me
[19:38:09] <bd808>	 That visibility tool is not scaling especially well as the use of the k8s cluster grows. It could probably use some caching love in the code base.
[19:38:23] <Majavah>	 bd808: are you trying to nerd snipe me
[19:39:00] <bd808>	 just stating facts Majavah, but if you are the sort to make things better I won't stop you ;)
[21:19:20] <legoktm>	 Majavah: if you have to ask if you're being nerd sniped, it's too late
[21:36:34] <bd808>	 legoktm: this is the way
[21:43:25] <legoktm>	 :D
[22:55:39] <Amir1>	 bstorm: hey, for when you have a bit time: https://gerrit.wikimedia.org/r/c/operations/puppet/+/661916 it's the last hiera() lookup in puppet files (some inside hiera yaml files are left but that's all)