[01:11:01] <wmcs-alerts>	 FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate fullstackd-20210112012248.admin-monitoring.eqiad1.wikimedia.cloud is about to expire in 14d 0h 15m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[02:44:32] <wikibugs>	 (03PS2) 10Reedy: Remove curl_close()/curl_multi_close() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1221217 (https://phabricator.wikimedia.org/T413538)
[03:00:03] <wikibugs>	 (03PS3) 10Reedy: Codesearch: Remove curl_close() [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1221217 (https://phabricator.wikimedia.org/T413538)
[04:21:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[06:16:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[08:42:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[10:17:11] <jinxer-wm>	 FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[11:21:31] <wikibugs>	 10Tool-community: Implementation of Authentication Feature - https://phabricator.wikimedia.org/T413550 (10Agamyasamuel) 03NEW
[12:10:05] <wikibugs>	 10Tool-curator: Improve title validation in frontend - https://phabricator.wikimedia.org/T413512#11485978 (10DaxServer) 05Open→03In progress
[12:16:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[12:16:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[12:42:59] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[16:42:59] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[19:15:35] <jinxer-wm>	 FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.992% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[19:35:35] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.966% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[20:42:59] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[20:58:55] <wikibugs>	 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11486202 (10Wyslijp16) +1