[01:11:01] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate fullstackd-20210112012248.admin-monitoring.eqiad1.wikimedia.cloud is about to expire in 12d 0h 15m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [05:25:25] 10Tool-wsindex, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11487094 (10Bodhisattwa) [07:52:20] 06cloud-services-team, 10PAWS: [Warning] Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413593 (10Leptictidium) 03NEW [07:55:18] 06cloud-services-team, 10PAWS: [Warning] Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413593#11487162 (10Leptictidium) [07:55:48] 06cloud-services-team, 10PAWS: [Warning] Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413593#11487163 (10Leptictidium) [08:41:43] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11487166 (10A_smart_kitten) [08:41:47] 06cloud-services-team, 10PAWS: [Warning] Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413593#11487169 (10A_smart_kitten) →14Duplicate dup:03T413510 [10:07:11] 06cloud-services-team, 10PAWS: Reason for logging into account - https://phabricator.wikimedia.org/T413597 (10Intobesa.bot) 03NEW [10:14:47] 06cloud-services-team, 10PAWS: Reason for logging into account - https://phabricator.wikimedia.org/T413597#11487222 (10A_smart_kitten) →14Duplicate dup:03T413510 [10:14:56] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11487224 (10A_smart_kitten) [10:23:32] 10Tool-curator: Improve title validation in frontend - https://phabricator.wikimedia.org/T413512#11487227 (10DaxServer) https://meta.wikimedia.org/wiki/Title_blacklist is actually required it seems, https://curator.toolforge.org/#/batches/422 [15:38:59] 06cloud-services-team, 10PAWS: [bug] - https://phabricator.wikimedia.org/T413607 (10Jrubashk) 03NEW [16:08:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:13:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:20:45] 06cloud-services-team, 10PAWS: [bug]  - https://phabricator.wikimedia.org/T413607#11487481 (10fnegri) →14Duplicate dup:03T413510 [16:20:46] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11487483 (10fnegri) [16:29:27] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11487487 (10fnegri) I had a quick look at this issue, the NFS server at paws-nfs-1.paws.eqiad1.wikimedia.cloud is failing to mount a shared volume: ` Dec 30 16:22:59 paws-nfs-1 mount[2779064]: mount.nfs: Faile... [17:34:37] (03open) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [17:44:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [17:51:52] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [18:02:35] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [18:16:41] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [18:30:06] (03PS1) 10Ladsgroup: Remove curl_close [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1221982 (https://phabricator.wikimedia.org/T413538) [18:41:16] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [18:48:14] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:24:19] (03CR) 10Reedy: "https://gerrit.wikimedia.org/r/c/labs/codesearch/+/1221217" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1221982 (https://phabricator.wikimedia.org/T413538) (owner: 10Ladsgroup) [19:56:12] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:06:47] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:11:30] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:24:24] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:37:30] (03update) 10don-vip: Draft: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:45:49] (03update) 10don-vip: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:46:27] (03CR) 10Ladsgroup: [C:03+2] "I will deploy it a bit later" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1221217 (https://phabricator.wikimedia.org/T413538) (owner: 10Reedy) [20:46:36] (03merge) 10don-vip: Umbra: improve titles [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/20 [20:47:23] (03Merged) 10jenkins-bot: Codesearch: Remove curl_close() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1221217 (https://phabricator.wikimedia.org/T413538) (owner: 10Reedy)