[01:59:40] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:33:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [03:59:25] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:02:43] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930518 (10SLyngshede-WMF) Shutdown Netbox on netbox-dev2003: ` slyngshede@netbox-dev2003:~$ sudo systemctl stop rq-netbox slyngshede@netbox-dev2003:~$ sudo systemctl stop uwsgi-netbox.service... [07:04:07] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930520 (10SLyngshede-WMF) ` List of databases Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | A... [07:04:34] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930523 (10SLyngshede-WMF) ` postgres=# \l postgres=# DROP DATABASE netbox; DROP DATABASE postgres=# ` [07:07:27] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930526 (10SLyngshede-WMF) Database restore ` gunzip < /srv/psql-all-dbs-latest.sql.gz | /usr/bin/psql ` Complains that some constraints already exists. This should be fine, as long the the co... [07:09:26] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930529 (10SLyngshede-WMF) Set password ` $ psql netbox ALTER USER netbox WITH PASSWORD '*********'; ` [07:11:01] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930530 (10SLyngshede-WMF) Restart services: ` sudo systemctl restart uwsgi-netbox.service sudo systemctl restart rq-netbox.service sudo systemctl status uwsgi-netbox.service sudo systemctl sta... [07:12:30] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930533 (10Volans) It's present twice in the dump: ` $ zgrep wireless_wirelesslin_interface_a_id_bc9e37fd_fk_dcim_inte /srv/psql-all-dbs-latest.sql.gz -- Name: wireless_wirelesslink wireless_wi... [07:21:37] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930571 (10SLyngshede-WMF) @Volans Thanks for checking :-) [07:24:53] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930582 (10SLyngshede-WMF) Deployment: ` cumin1002:~$ sudo cookbook sre.deploy.python-code -t T397300 -r 'Release v4.0.11 to netbox-next' -u netbox netbox 'A:netbox-canary' ` [07:26:56] 10netbox, 06Infrastructure-Foundations: Upgrade Netbox to version 4.0.11 - https://phabricator.wikimedia.org/T397300#10930588 (10ops-monitoring-bot) Deployed netbox to netbox-dev2003.codfw.wmnet with reason: Release v4.0.11 to netbox-next - slyngshede@cumin1002 - T397300 [07:33:29] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:03:34] FIRING: DiskSpace: Disk space cumin2002:9100:/ 5.998% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cumin2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:18:34] RESOLVED: DiskSpace: Disk space cumin2002:9100:/ 5.998% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cumin2002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:40:33] Morning. I have a query around LDAP groups and CAS/OIDC authentication. It has arisen because I have lost my Superset admin privileges. [08:41:16] I re-added myself to this group yesterday, using `ldapvi` on seaborgium. https://ldap.toolforge.org/group/superset-admins [08:44:18] Superset is checking for these groups returned from the IDP servers: [08:44:25] https://www.irccloud.com/pastebin/7oDyFfOt/ [08:46:01] the CAS config for superset_k8s and superset_next_k8s only has cn=wmf and cn=nda configured, this this work in the past [08:46:10] the CAS config for superset_k8s and superset_next_k8s only has cn=wmf and cn=nda configured, did this work in the past? [08:47:02] But when I log out and back in, I'm not getting this group, although I used to. Is there something I need to do to Apereo/Bitu after changing the groups? [08:47:35] Yes, this is currently working for other members of DPE/SRE. They are assigned the Admin role. [08:48:10] if you go to https://idp.wikimedia.org/login, do you see superset-admins under memberOf? [08:48:30] if not, then you probably have a remember me session which persists [08:48:35] I think that the CAS config is correct, because all members of `wmf` and `nda` should be permitted to use Superset. It's an additional layer of role mapping. Checking now. [08:48:52] you can force a refresh via https://idp.wikimedia.org/logout [08:48:57] and the log in again [08:49:05] No, I don't see that group. OK, logging out now. [08:49:33] Yes, I see it now. [08:50:23] And it's fixed in Superset. Thanks Moritz. I feel a bit daft now :-) [09:08:31] as with all things in computers it's either caches or DNS :-) [09:09:30] or MTU