[00:05:01] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27005MiB (5% inode=99%); [00:06:03] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26826MiB (5% inode=99%); [00:07:08] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [00:08:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750115300000&orgId=1&to=1750118933359 [00:09:01] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 55048MiB (12% inode=99%); [00:10:03] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55340MiB (12% inode=99%); [00:13:08] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 34% [00:18:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750115720000&orgId=1&to=1750119320000 [00:23:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2f2993ddb560d966ac0ce5e1bd2db5f447d323b8 [00:23:16] 02puppet/03main 07CosmicAlpha 032f2993d Update prometheus_jobqueue_stats to timer [00:25:58] !log remove root crontab on test151 (all crons on test151 are migrated to systemd timers) [00:26:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:46] !log remove root crontab on db (all crons on db are migrated to systemd timers) [00:27:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:42] !log remove root crontab on phorge171 (all crons on phorge171 are migrated to systemd timers) [00:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e163948f36359c1b48d48df0b62923b654f86ae6 [00:33:31] 02puppet/03main 07CosmicAlpha 03e163948 Update remove_icinga2_perfdata to timer [00:38:44] !log remove root crontab on mon181 (all crons on mon181 are migrated to systemd timers) [00:38:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750117460000&orgId=1&to=1750121093355 [00:46:53] [02puppet] 07Universal-Omega created 03pywikibot-timer (+1 new commit) 13https://github.com/miraheze/puppet/commit/9f4b04829ef1 [00:46:53] 02puppet/03pywikibot-timer 07CosmicAlpha 039f4b048 pywikibot: use systemd timers instead of crons [00:46:58] [02puppet] 07Universal-Omega opened pull request #4401: pywikibot: use systemd timers instead of crons (03main...03pywikibot-timer) 13https://github.com/miraheze/puppet/pull/4401 [00:47:03] [02puppet] 07coderabbitai[bot] commented on pull request #4401:
[…] 13https://github.com/miraheze/puppet/pull/4401#issuecomment-2978579055 [00:49:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750117700000&orgId=1&to=1750121300000 [00:55:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03pywikibot-timer 13https://github.com/miraheze/puppet/commit/d14d79c7de49ed3f612d49e08bf5488503653f38 [00:55:07] 02puppet/03pywikibot-timer 07CosmicAlpha 03d14d79c Update [00:55:48] [02puppet] 07github-actions[bot] pushed 1 new commit to 03pywikibot-timer 13https://github.com/miraheze/puppet/commit/4794daa50b241267eca65cdae0f0bd4a3bafe520 [00:55:48] 02puppet/03pywikibot-timer 07github-actions 034794daa CI: lint puppet code to standards… [00:58:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03pywikibot-timer 13https://github.com/miraheze/puppet/commit/881065359f6229be470cb7b13c3795f027221ecd [00:58:16] 02puppet/03pywikibot-timer 07CosmicAlpha 038810653 - [01:09:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750118960000&orgId=1&to=1750122593353 [01:12:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/813fe4734a09a372715d4e464bd9fc4fe7ca7fa8 [01:12:56] 02puppet/03main 07CosmicAlpha 03813fe47 systemd: add monitoring for timers… [01:12:58] [02puppet] 07Universal-Omega merged pull request #4393: systemd: add monitoring for timers (03main...03systemd-monitor) 13https://github.com/miraheze/puppet/pull/4393 [01:13:00] [02puppet] 07Universal-Omega 04deleted 03systemd-monitor at 0363f4e1f 13https://api.github.com/repos/miraheze/puppet/commit/63f4e1f [01:15:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9cb6f343cab8d00f15f9e5bab07ef2959ec4ff0b [01:15:51] 02puppet/03main 07CosmicAlpha 039cb6f34 Fix [01:16:37] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:17:25] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:18:04] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:19:25] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [01:19:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750119560000&orgId=1&to=1750123160000 [01:22:47] PROBLEM - cp191 Check unit status of puppet-agent-timer on cp191 is UNKNOWN: NRPE: Unable to read output [01:22:47] PROBLEM - cloud17 Check unit status of puppet-agent-timer on cloud17 is UNKNOWN: NRPE: Unable to read output [01:22:47] PROBLEM - cloud15 Check unit status of puppet-agent-timer on cloud15 is UNKNOWN: NRPE: Unable to read output [01:22:52] PROBLEM - mw152 Check unit status of puppet-agent-timer on mw152 is UNKNOWN: NRPE: Unable to read output [01:22:54] PROBLEM - mw192 Check unit status of puppet-agent-timer on mw192 is UNKNOWN: NRPE: Unable to read output [01:22:55] PROBLEM - bast181 Check unit status of puppet-agent-timer on bast181 is UNKNOWN: NRPE: Unable to read output [01:22:56] PROBLEM - mw151 Check unit status of puppet-agent-timer on mw151 is UNKNOWN: NRPE: Unable to read output [01:22:56] PROBLEM - puppet181 Check unit status of puppet-agent-timer on puppet181 is UNKNOWN: NRPE: Unable to read output [01:22:57] PROBLEM - db181 Check unit status of db-backups on db181 is UNKNOWN: NRPE: Unable to read output [01:22:57] PROBLEM - mem151 Check unit status of puppet-agent-timer on mem151 is UNKNOWN: NRPE: Unable to read output [01:22:57] PROBLEM - puppet181 Check unit status of backups-private on puppet181 is UNKNOWN: NRPE: Unable to read output [01:22:58] PROBLEM - puppet181 Check unit status of updatesfs on puppet181 is UNKNOWN: NRPE: Unable to read output [01:23:00] PROBLEM - puppet181 Check unit status of backups-sslkeys on puppet181 is UNKNOWN: NRPE: Unable to read output [01:23:00] PROBLEM - kafka181 Check unit status of zookeeper-cleanup on kafka181 is UNKNOWN: NRPE: Unable to read output [01:23:01] PROBLEM - mem161 Check unit status of puppet-agent-timer on mem161 is UNKNOWN: NRPE: Unable to read output [01:23:01] PROBLEM - db172 Check unit status of db-backups on db172 is UNKNOWN: NRPE: Unable to read output [01:23:01] PROBLEM - mw181 Check unit status of puppet-agent-timer on mw181 is UNKNOWN: NRPE: Unable to read output [01:23:01] PROBLEM - puppet181 Check unit status of geoipupdate on puppet181 is UNKNOWN: NRPE: Unable to read output [01:23:01] PROBLEM - db171 Check unit status of db-backups on db171 is UNKNOWN: NRPE: Unable to read output [01:23:02] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:23:02] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:23:03] PROBLEM - mw193 Check unit status of puppet-agent-timer on mw193 is UNKNOWN: NRPE: Unable to read output [01:23:14] PROBLEM - db171 Check unit status of mhglobal-db-backups-weekly on db171 is UNKNOWN: NRPE: Unable to read output [01:23:15] PROBLEM - mem191 Check unit status of puppet-agent-timer on mem191 is UNKNOWN: NRPE: Unable to read output [01:23:15] PROBLEM - mwtask151 Check unit status of puppet-agent-timer on mwtask151 is UNKNOWN: NRPE: Unable to read output [01:23:16] PROBLEM - os201 Check unit status of puppet-agent-timer on os201 is UNKNOWN: NRPE: Unable to read output [01:23:17] PROBLEM - cp161 Check unit status of puppet-agent-timer on cp161 is UNKNOWN: NRPE: Unable to read output [01:23:17] PROBLEM - mattermost1 Check unit status of puppet-agent-timer on mattermost1 is UNKNOWN: NRPE: Unable to read output [01:23:20] PROBLEM - mw203 Check unit status of puppet-agent-timer on mw203 is UNKNOWN: NRPE: Unable to read output [01:23:21] PROBLEM - graylog161 Check unit status of puppet-agent-timer on graylog161 is UNKNOWN: NRPE: Unable to read output [01:23:21] PROBLEM - os151 Check unit status of puppet-agent-timer on os151 is UNKNOWN: NRPE: Unable to read output [01:23:22] PROBLEM - db181 Check unit status of puppet-agent-timer on db181 is UNKNOWN: NRPE: Unable to read output [01:23:23] PROBLEM - ns1 Check unit status of puppet-agent-timer on ns1 is UNKNOWN: NRPE: Unable to read output [01:23:23] PROBLEM - db161 Check unit status of db-backups on db161 is UNKNOWN: NRPE: Unable to read output [01:23:24] PROBLEM - cp171 Check unit status of varnish-requests-exporter on cp171 is UNKNOWN: NRPE: Unable to read output [01:23:24] PROBLEM - mw162 Check unit status of puppet-agent-timer on mw162 is UNKNOWN: NRPE: Unable to read output [01:23:25] PROBLEM - matomo151 Check unit status of matomo-optimize-january on matomo151 is UNKNOWN: NRPE: Unable to read output [01:23:25] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:23:26] PROBLEM - db172 Check unit status of puppet-agent-timer on db172 is UNKNOWN: NRPE: Unable to read output [01:23:27] PROBLEM - db182 Check unit status of reports-db-backups-weekly on db182 is UNKNOWN: NRPE: Unable to read output [01:23:27] PROBLEM - mw172 Check unit status of puppet-agent-timer on mw172 is UNKNOWN: NRPE: Unable to read output [01:23:29] PROBLEM - cloud16 Check unit status of puppet-agent-timer on cloud16 is UNKNOWN: NRPE: Unable to read output [01:23:29] PROBLEM - mw182 Check unit status of puppet-agent-timer on mw182 is UNKNOWN: NRPE: Unable to read output [01:23:30] PROBLEM - mattermost1 Check unit status of mattermost-data-backup on mattermost1 is UNKNOWN: NRPE: Unable to read output [01:23:31] PROBLEM - puppet181 Check unit status of puppetdb-backup on puppet181 is UNKNOWN: NRPE: Unable to read output [01:23:31] PROBLEM - ns1 Check unit status of prometheus_gdnsd_stats on ns1 is UNKNOWN: NRPE: Unable to read output [01:23:31] PROBLEM - mwtask171 Check unit status of puppet-agent-timer on mwtask171 is UNKNOWN: NRPE: Unable to read output [01:23:33] PROBLEM - db171 Check unit status of puppet-agent-timer on db171 is UNKNOWN: NRPE: Unable to read output [01:23:34] PROBLEM - cloud18 Check unit status of puppet-agent-timer on cloud18 is UNKNOWN: NRPE: Unable to read output [01:23:37] PROBLEM - kafka181 Check unit status of puppet-agent-timer on kafka181 is UNKNOWN: NRPE: Unable to read output [01:23:37] PROBLEM - ldap171 Check unit status of puppet-agent-timer on ldap171 is UNKNOWN: NRPE: Unable to read output [01:23:38] PROBLEM - matomo151 Check unit status of puppet-agent-timer on matomo151 is UNKNOWN: NRPE: Unable to read output [01:23:39] PROBLEM - mw163 Check unit status of puppet-agent-timer on mw163 is UNKNOWN: NRPE: Unable to read output [01:23:39] PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is UNKNOWN: NRPE: Unable to read output [01:23:40] PROBLEM - matomo151 Check unit status of matomo-optimize-last2 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:23:40] PROBLEM - db151 Check unit status of db-backups on db151 is UNKNOWN: NRPE: Unable to read output [01:23:40] PROBLEM - mon181 Check unit status of remove_icinga2_perfdata on mon181 is UNKNOWN: NRPE: Unable to read output [01:23:41] PROBLEM - db182 Check unit status of puppet-agent-timer on db182 is UNKNOWN: NRPE: Unable to read output [01:26:30] PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is UNKNOWN: NRPE: Unable to read output [01:26:31] PROBLEM - swiftobject151 Check unit status of puppet-agent-timer on swiftobject151 is UNKNOWN: NRPE: Unable to read output [01:26:32] PROBLEM - reports171 Check unit status of task-scheduler on reports171 is UNKNOWN: NRPE: Unable to read output [01:26:33] PROBLEM - os191 Check unit status of puppet-agent-timer on os191 is UNKNOWN: NRPE: Unable to read output [01:26:36] PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is UNKNOWN: NRPE: Unable to read output [01:26:37] PROBLEM - swiftobject171 Check unit status of disable-rsync on swiftobject171 is UNKNOWN: NRPE: Unable to read output [01:26:37] PROBLEM - swiftobject171 Check unit status of puppet-agent-timer on swiftobject171 is UNKNOWN: NRPE: Unable to read output [01:26:38] PROBLEM - swiftproxy171 Check unit status of puppet-agent-timer on swiftproxy171 is UNKNOWN: NRPE: Unable to read output [01:26:39] PROBLEM - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:26:39] PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is UNKNOWN: NRPE: Unable to read output [01:26:40] PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is UNKNOWN: NRPE: Unable to read output [01:26:45] PROBLEM - phorge171 Check unit status of puppet-agent-timer on phorge171 is UNKNOWN: NRPE: Unable to read output [01:26:46] PROBLEM - ns2 Check unit status of puppet-agent-timer on ns2 is UNKNOWN: NRPE: Unable to read output [01:26:46] PROBLEM - os161 Check unit status of puppet-agent-timer on os161 is UNKNOWN: NRPE: Unable to read output [01:26:50] PROBLEM - swiftobject201 Check unit status of puppet-agent-timer on swiftobject201 is UNKNOWN: NRPE: Unable to read output [01:26:50] PROBLEM - swiftobject201 Check unit status of disable-rsync on swiftobject201 is UNKNOWN: NRPE: Unable to read output [01:26:50] PROBLEM - swiftproxy171 Check unit status of swift-account-stats_admin:admin on swiftproxy171 is UNKNOWN: NRPE: Unable to read output [01:26:52] PROBLEM - swiftobject191 Check unit status of puppet-agent-timer on swiftobject191 is UNKNOWN: NRPE: Unable to read output [01:26:52] PROBLEM - prometheus151 Check unit status of puppet-agent-timer on prometheus151 is UNKNOWN: NRPE: Unable to read output [01:26:55] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-parsercache on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:26:55] PROBLEM - swiftobject181 Check unit status of disable-rsync on swiftobject181 is UNKNOWN: NRPE: Unable to read output [01:26:56] PROBLEM - mwtask181 Check unit status of mediawiki_job_generate-sitemap-index on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:26:57] PROBLEM - swiftobject181 Check unit status of puppet-agent-timer on swiftobject181 is UNKNOWN: NRPE: Unable to read output [01:26:57] PROBLEM - swiftproxy171 Check unit status of swift-account-stats_swift:dispersion on swiftproxy171 is UNKNOWN: NRPE: Unable to read output [01:26:58] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:26:59] PROBLEM - mwtask181 Check unit status of puppet-agent-timer on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:26:59] PROBLEM - swiftac171 Check unit status of puppet-agent-timer on swiftac171 is UNKNOWN: NRPE: Unable to read output [01:27:00] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-loginnotify on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:27:00] PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is UNKNOWN: NRPE: Unable to read output [01:27:00] PROBLEM - swiftobject151 Check unit status of disable-rsync on swiftobject151 is UNKNOWN: NRPE: Unable to read output [01:27:00] PROBLEM - swiftac171 Check unit status of backups-swift-account-container on swiftac171 is UNKNOWN: NRPE: Unable to read output [01:27:01] PROBLEM - mwtask181 Check unit status of mediawiki_job_generate-sitemaps on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:27:01] PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is UNKNOWN: NRPE: Unable to read output [01:27:04] PROBLEM - swiftobject161 Check unit status of disable-rsync on swiftobject161 is UNKNOWN: NRPE: Unable to read output [01:27:05] PROBLEM - swiftobject191 Check unit status of disable-rsync on swiftobject191 is UNKNOWN: NRPE: Unable to read output [01:27:06] PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is UNKNOWN: NRPE: Unable to read output [01:27:09] PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is UNKNOWN: NRPE: Unable to read output [01:27:09] PROBLEM - swiftproxy171 Check unit status of swift-account-stats_mw:media on swiftproxy171 is UNKNOWN: NRPE: Unable to read output [01:27:09] PROBLEM - ns2 Check unit status of prometheus_gdnsd_stats on ns2 is UNKNOWN: NRPE: Unable to read output [01:27:10] PROBLEM - swiftproxy161 Check unit status of puppet-agent-timer on swiftproxy161 is UNKNOWN: NRPE: Unable to read output [01:27:10] PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is UNKNOWN: NRPE: Unable to read output [01:27:10] PROBLEM - os162 Check unit status of puppet-agent-timer on os162 is UNKNOWN: NRPE: Unable to read output [01:27:11] PROBLEM - phorge171 Check unit status of phorge-backup on phorge171 is UNKNOWN: NRPE: Unable to read output [01:27:11] PROBLEM - swiftobject161 Check unit status of puppet-agent-timer on swiftobject161 is UNKNOWN: NRPE: Unable to read output [01:27:23] PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is UNKNOWN: NRPE: Unable to read output [01:27:24] PROBLEM - test151 Check unit status of puppet-agent-timer on test151 is UNKNOWN: NRPE: Unable to read output [01:27:25] PROBLEM - mw202 Check unit status of puppet-agent-timer on mw202 is UNKNOWN: NRPE: Unable to read output [01:27:25] PROBLEM - rdb151 Check unit status of puppet-agent-timer on rdb151 is UNKNOWN: NRPE: Unable to read output [01:27:25] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-expired-blocks on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:27:26] PROBLEM - mwtask181 Check unit status of mediawiki_job_cleanup-upload-stash on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:27:28] PROBLEM - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:27:29] PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is UNKNOWN: NRPE: Unable to read output [01:27:29] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is UNKNOWN: NRPE: Unable to read output [01:27:29] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:27:29] PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is UNKNOWN: NRPE: Unable to read output [01:27:30] PROBLEM - reports171 Check unit status of puppet-agent-timer on reports171 is UNKNOWN: NRPE: Unable to read output [01:31:21] !log [somerandomdeveloper@mwtask171] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=battlenationswiki --i=1 --u=Miraheze_Operations --r='[[phab:T13542|Requested]]' /home/somerandomdeveloper/deletebatch.txt (END - exit=0) [01:31:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:20] [02python-functions] 07dependabot[bot] created 03dependabot/pip/dot-github/mypy-1.16.1 (+1 new commit) 13https://github.com/miraheze/python-functions/commit/b0b116b33331 [01:33:20] 02python-functions/03dependabot/pip/dot-github/mypy-1.16.1 07dependabot[bot] 03b0b116b build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github… [01:33:22] [02python-functions] 07dependabot[bot] added the label 'python' to pull request #151 (build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github) 13https://github.com/miraheze/python-functions/pull/151 [01:33:24] [02python-functions] 07dependabot[bot] added the label 'dependencies' to pull request #151 (build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github) 13https://github.com/miraheze/python-functions/pull/151 [01:33:26] [02python-functions] 07dependabot[bot] opened pull request #151: build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github (03main...03dependabot/pip/dot-github/mypy-1.16.1) 13https://github.com/miraheze/python-functions/pull/151 [01:33:28] [02python-functions] 07dependabot[bot] added the label 'dependencies' to pull request #151 (build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github) 13https://github.com/miraheze/python-functions/pull/151 [01:33:30] [02python-functions] 07dependabot[bot] added the label 'python' to pull request #151 (build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github) 13https://github.com/miraheze/python-functions/pull/151 [01:33:32] [02python-functions] 07coderabbitai[bot] commented on pull request #151: --- […] 13https://github.com/miraheze/python-functions/pull/151#issuecomment-2978642497 [01:34:10] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cf1013360df83979cc301c607404f1ba8974ebeb [01:34:10] 02puppet/03main 07CosmicAlpha 03cf10133 Fix [01:35:19] RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl [01:35:30] RECOVERY - puppet181 Check unit status of remove_old_puppet_reports on puppet181 is OK: OK: Status of the systemd unit remove_old_puppet_reports [01:36:21] RECOVERY - cloud15 Check unit status of puppet-agent-timer on cloud15 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:22] PROBLEM - db171 Check unit status of db-backups on db171 is CRITICAL: CRITICAL: Status of the systemd unit db-backups [01:36:24] RECOVERY - puppet181 Check unit status of backups-private on puppet181 is OK: OK: Status of the systemd unit backups-private [01:36:25] RECOVERY - puppet181 Check unit status of geoipupdate on puppet181 is OK: OK: Status of the systemd unit geoipupdate [01:36:26] RECOVERY - kafka181 Check unit status of zookeeper-cleanup on kafka181 is OK: OK: Status of the systemd unit zookeeper-cleanup [01:36:28] RECOVERY - ldap171 Check unit status of restart_slapd on ldap171 is OK: OK: Status of the systemd unit restart_slapd [01:36:30] RECOVERY - puppet181 Check unit status of puppet-agent-timer on puppet181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:31] RECOVERY - bast161 Check unit status of puppet-agent-timer on bast161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:31] RECOVERY - cp171 Check unit status of haproxy_stek_job on cp171 is OK: OK: Status of the systemd unit haproxy_stek_job [01:36:32] RECOVERY - prometheus151 Check unit status of puppet-agent-timer on prometheus151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:33] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [01:36:34] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [01:36:34] RECOVERY - cloud17 Check unit status of puppet-agent-timer on cloud17 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:34] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [01:36:38] RECOVERY - db182 Check unit status of icingaweb2-db-backups-fortnightly on db182 is OK: OK: Status of the systemd unit icingaweb2-db-backups-fortnightly [01:36:40] RECOVERY - graylog161 Check unit status of puppet-agent-timer on graylog161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:41] RECOVERY - bast181 Check unit status of puppet-agent-timer on bast181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:42] RECOVERY - db181 Check unit status of db-backups on db181 is OK: OK: Status of the systemd unit db-backups [01:36:45] PROBLEM - graylog161 Check unit status of prometheus_graylog_stats on graylog161 is CRITICAL: CRITICAL: Status of the systemd unit prometheus_graylog_stats [01:36:46] RECOVERY - eventgate181 Check unit status of puppet-agent-timer on eventgate181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:47] RECOVERY - puppet181 Check unit status of updatesfs on puppet181 is OK: OK: Status of the systemd unit updatesfs [01:36:49] RECOVERY - cp191 Check unit status of varnish-requests-exporter on cp191 is OK: OK: Status of the systemd unit varnish-requests-exporter [01:36:49] RECOVERY - db172 Check unit status of db-backups on db172 is OK: OK: Status of the systemd unit db-backups [01:36:49] RECOVERY - cp191 Check unit status of haproxy_stek_job on cp191 is OK: OK: Status of the systemd unit haproxy_stek_job [01:36:49] RECOVERY - puppet181 Check unit status of backups-sslkeys on puppet181 is OK: OK: Status of the systemd unit backups-sslkeys [01:36:50] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-1 [01:36:52] RECOVERY - mattermost1 Check unit status of mattermost-db-backup on mattermost1 is OK: OK: Status of the systemd unit mattermost-db-backup [01:36:53] RECOVERY - cp171 Check unit status of varnish-requests-exporter on cp171 is OK: OK: Status of the systemd unit varnish-requests-exporter [01:36:53] RECOVERY - db171 Check unit status of mhglobal-db-backups-weekly on db171 is OK: OK: Status of the systemd unit mhglobal-db-backups-weekly [01:36:54] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [01:36:55] RECOVERY - cloud19 Check unit status of puppet-agent-timer on cloud19 is OK: OK: Status of the systemd unit puppet-agent-timer [01:36:56] RECOVERY - db161 Check unit status of db-backups on db161 is OK: OK: Status of the systemd unit db-backups [01:36:59] RECOVERY - matomo151 Check unit status of matomo-optimize-january on matomo151 is OK: OK: Status of the systemd unit matomo-optimize-january [01:37:00] RECOVERY - db172 Check unit status of puppet-agent-timer on db172 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:02] RECOVERY - bots171 Check unit status of puppet-agent-timer on bots171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:02] RECOVERY - cp161 Check unit status of puppet-agent-timer on cp161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:02] RECOVERY - puppet181 Check unit status of puppetdb-backup on puppet181 is OK: OK: Status of the systemd unit puppetdb-backup [01:37:02] RECOVERY - cloud16 Check unit status of puppet-agent-timer on cloud16 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:05] RECOVERY - mattermost1 Check unit status of puppet-agent-timer on mattermost1 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:06] RECOVERY - mattermost1 Check unit status of mattermost-data-backup on mattermost1 is OK: OK: Status of the systemd unit mattermost-data-backup [01:37:07] RECOVERY - db182 Check unit status of reports-db-backups-weekly on db182 is OK: OK: Status of the systemd unit reports-db-backups-weekly [01:37:08] RECOVERY - db181 Check unit status of puppet-agent-timer on db181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:18] RECOVERY - cloud18 Check unit status of puppet-agent-timer on cloud18 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:21] RECOVERY - ldap171 Check unit status of puppet-agent-timer on ldap171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:21] RECOVERY - db171 Check unit status of puppet-agent-timer on db171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:25] [02python-functions] 07github-merge-queue[bot] created 03gh-readonly-queue/main/pr-151-aa07e26f364761c93166af84bfcc3292d94ab989 (+1 new commit) 13https://github.com/miraheze/python-functions/commit/67e1858c7540 [01:37:26] 02python-functions/03gh-readonly-queue/main/pr-151-aa07e26f364761c93166af84bfcc3292d94ab989 07dependabot[bot] 0367e1858 build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github (#151)… [01:37:27] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-2 [01:37:27] miraheze/python-functions - dependabot[bot] the build passed. [01:37:29] RECOVERY - cp161 Check unit status of varnish-requests-exporter on cp161 is OK: OK: Status of the systemd unit varnish-requests-exporter [01:37:32] RECOVERY - db151 Check unit status of db-backups on db151 is OK: OK: Status of the systemd unit db-backups [01:37:35] RECOVERY - kafka181 Check unit status of puppet-agent-timer on kafka181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:36] RECOVERY - db182 Check unit status of puppet-agent-timer on db182 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:37] RECOVERY - db161 Check unit status of puppet-agent-timer on db161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:49] RECOVERY - matomo151 Check unit status of matomo-optimize-last2 on matomo151 is OK: OK: Status of the systemd unit matomo-optimize-last2 [01:37:53] RECOVERY - changeprop201 Check unit status of puppet-agent-timer on changeprop201 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:56] RECOVERY - mw183 Check unit status of puppet-agent-timer on mw183 is OK: OK: Status of the systemd unit puppet-agent-timer [01:37:56] RECOVERY - cloud20 Check unit status of puppet-agent-timer on cloud20 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:01] RECOVERY - cp191 Check unit status of puppet-agent-timer on cp191 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:04] RECOVERY - cp171 Check unit status of puppet-agent-timer on cp171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:07] RECOVERY - matomo151 Check unit status of puppet-agent-timer on matomo151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:07] RECOVERY - mem151 Check unit status of puppet-agent-timer on mem151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:09] RECOVERY - db182 Check unit status of icinga-db-backups-fortnightly on db182 is OK: OK: Status of the systemd unit icinga-db-backups-fortnightly [01:38:11] RECOVERY - mw152 Check unit status of puppet-agent-timer on mw152 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:16] RECOVERY - db151 Check unit status of puppet-agent-timer on db151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:18] RECOVERY - mw191 Check unit status of puppet-agent-timer on mw191 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:20] RECOVERY - ns1 Check unit status of prometheus_gdnsd_stats on ns1 is OK: OK: Status of the systemd unit prometheus_gdnsd_stats [01:38:26] RECOVERY - mw153 Check unit status of puppet-agent-timer on mw153 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:31] RECOVERY - mw181 Check unit status of puppet-agent-timer on mw181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:33] RECOVERY - mw192 Check unit status of puppet-agent-timer on mw192 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:33] RECOVERY - mem161 Check unit status of puppet-agent-timer on mem161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:35] RECOVERY - ns2 Check unit status of puppet-agent-timer on ns2 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:37] RECOVERY - mwtask181 Check unit status of puppet-agent-timer on mwtask181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:39] RECOVERY - mw173 Check unit status of puppet-agent-timer on mw173 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:41] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_update-special-pages [01:38:41] RECOVERY - mw193 Check unit status of puppet-agent-timer on mw193 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:42] RECOVERY - ns1 Check unit status of puppet-agent-timer on ns1 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:42] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-parsercache on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache [01:38:43] RECOVERY - mwtask161 Check unit status of puppet-agent-timer on mwtask161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:45] RECOVERY - mwtask181 Check unit status of mediawiki_job_generate-sitemaps on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps [01:38:46] RECOVERY - mem191 Check unit status of puppet-agent-timer on mem191 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:47] RECOVERY - mem201 Check unit status of puppet-agent-timer on mem201 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:47] RECOVERY - mw151 Check unit status of puppet-agent-timer on mw151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:48] RECOVERY - swiftobject191 Check unit status of disable-rsync on swiftobject191 is OK: OK: Status of the systemd unit disable-rsync [01:38:50] RECOVERY - mw203 Check unit status of puppet-agent-timer on mw203 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:51] RECOVERY - swiftobject181 Check unit status of disable-rsync on swiftobject181 is OK: OK: Status of the systemd unit disable-rsync [01:38:52] RECOVERY - ns2 Check unit status of prometheus_gdnsd_stats on ns2 is OK: OK: Status of the systemd unit prometheus_gdnsd_stats [01:38:52] RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media [01:38:53] RECOVERY - mwtask181 Check unit status of mediawiki_job_generate-sitemap-index on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemap-index [01:38:54] RECOVERY - swiftac171 Check unit status of puppet-agent-timer on swiftac171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:54] RECOVERY - swiftproxy171 Check unit status of swift-account-stats_mw:media on swiftproxy171 is OK: OK: Status of the systemd unit swift-account-stats_mw:media [01:38:56] RECOVERY - reports171 Check unit status of puppet-agent-timer on reports171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:57] RECOVERY - mwtask171 Check unit status of puppet-agent-timer on mwtask171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:58] RECOVERY - mw171 Check unit status of puppet-agent-timer on mw171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:38:59] RECOVERY - rdb151 Check unit status of puppet-agent-timer on rdb151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:00] RECOVERY - mwtask151 Check unit status of puppet-agent-timer on mwtask151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:01] RECOVERY - os151 Check unit status of puppet-agent-timer on os151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:01] RECOVERY - swiftproxy161 Check unit status of puppet-agent-timer on swiftproxy161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:02] RECOVERY - mw161 Check unit status of puppet-agent-timer on mw161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:04] RECOVERY - mw172 Check unit status of puppet-agent-timer on mw172 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:07] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-expired-blocks on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks [01:39:07] RECOVERY - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis [01:39:08] RECOVERY - os162 Check unit status of puppet-agent-timer on os162 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:08] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-checkuser on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser [01:39:09] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-abusefilter on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter [01:39:13] RECOVERY - mw162 Check unit status of puppet-agent-timer on mw162 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:14] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_update-statistics [01:39:14] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table [01:39:14] RECOVERY - mw182 Check unit status of puppet-agent-timer on mw182 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:14] RECOVERY - mwtask181 Check unit status of mediawiki_job_refreshlinks on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks [01:39:16] RECOVERY - mw163 Check unit status of puppet-agent-timer on mw163 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:20] RECOVERY - mwtask181 Check unit status of mediawiki_job_cleanup-upload-stash on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash [01:39:20] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [01:39:22] RECOVERY - mw202 Check unit status of puppet-agent-timer on mw202 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:27] RECOVERY - mon181 Check unit status of grafana-backup on mon181 is OK: OK: Status of the systemd unit grafana-backup [01:39:42] RECOVERY - mw201 Check unit status of puppet-agent-timer on mw201 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:48] RECOVERY - mon181 Check unit status of puppet-agent-timer on mon181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:52] RECOVERY - swiftobject171 Check unit status of puppet-agent-timer on swiftobject171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:39:58] RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify [01:39:58] RECOVERY - mon181 Check unit status of remove_icinga2_perfdata on mon181 is OK: OK: Status of the systemd unit remove_icinga2_perfdata [01:39:58] RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia [01:40:04] RECOVERY - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table [01:40:11] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-loginnotify on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify [01:40:11] RECOVERY - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia [01:40:11] RECOVERY - swiftobject171 Check unit status of disable-rsync on swiftobject171 is OK: OK: Status of the systemd unit disable-rsync [01:40:13] RECOVERY - phorge171 Check unit status of puppet-agent-timer on phorge171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:14] RECOVERY - swiftobject201 Check unit status of puppet-agent-timer on swiftobject201 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:15] RECOVERY - swiftproxy171 Check unit status of swift-account-stats_swift:dispersion on swiftproxy171 is OK: OK: Status of the systemd unit swift-account-stats_swift:dispersion [01:40:16] RECOVERY - os161 Check unit status of puppet-agent-timer on os161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:19] RECOVERY - swiftobject151 Check unit status of puppet-agent-timer on swiftobject151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:20] RECOVERY - swiftproxy171 Check unit status of swift-account-stats_admin:admin on swiftproxy171 is OK: OK: Status of the systemd unit swift-account-stats_admin:admin [01:40:21] RECOVERY - swiftobject191 Check unit status of puppet-agent-timer on swiftobject191 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:22] PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_update-statistics [01:40:24] RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis [01:40:25] RECOVERY - swiftobject161 Check unit status of disable-rsync on swiftobject161 is OK: OK: Status of the systemd unit disable-rsync [01:40:25] RECOVERY - swiftobject181 Check unit status of puppet-agent-timer on swiftobject181 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:26] RECOVERY - reports171 Check unit status of task-scheduler on reports171 is OK: OK: Status of the systemd unit task-scheduler [01:40:26] RECOVERY - phorge171 Check unit status of phorge-backup on phorge171 is OK: OK: Status of the systemd unit phorge-backup [01:40:28] RECOVERY - os201 Check unit status of puppet-agent-timer on os201 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:28] RECOVERY - os191 Check unit status of puppet-agent-timer on os191 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:29] RECOVERY - swiftobject201 Check unit status of disable-rsync on swiftobject201 is OK: OK: Status of the systemd unit disable-rsync [01:40:31] RECOVERY - swiftac171 Check unit status of backups-swift-account-container on swiftac171 is OK: OK: Status of the systemd unit backups-swift-account-container [01:40:31] RECOVERY - swiftproxy171 Check unit status of puppet-agent-timer on swiftproxy171 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:31] RECOVERY - swiftobject161 Check unit status of puppet-agent-timer on swiftobject161 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:32] RECOVERY - test151 Check unit status of prometheus_jobqueue_stats on test151 is OK: OK: Status of the systemd unit prometheus_jobqueue_stats [01:40:38] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats [01:40:38] RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks [01:40:41] RECOVERY - test151 Check unit status of puppet-agent-timer on test151 is OK: OK: Status of the systemd unit puppet-agent-timer [01:40:41] RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter [01:40:43] RECOVERY - swiftobject151 Check unit status of disable-rsync on swiftobject151 is OK: OK: Status of the systemd unit disable-rsync [01:40:43] RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps [01:40:47] RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser [01:40:48] RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks [01:40:50] RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache [01:40:58] RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash [01:41:07] [02python-functions] 07github-merge-queue[bot] merged 03gh-readonly-queue/main/pr-151-aa07e26f364761c93166af84bfcc3292d94ab989 into 03main 13https://github.com/miraheze/python-functions/compare/aa07e26f3647...67e1858c7540 [01:41:07] [02python-functions] 07github-actions[bot] merged 07dependabot[bot]'s pull request #151: build(deps): bump mypy from 1.16.0 to 1.16.1 in /.github (03main...03dependabot/pip/dot-github/mypy-1.16.1) 13https://github.com/miraheze/python-functions/pull/151 [01:41:08] [02python-functions] 07github-merge-queue[bot] 04deleted 03gh-readonly-queue/main/pr-151-aa07e26f364761c93166af84bfcc3292d94ab989 at 0367e1858 13https://api.github.com/repos/miraheze/python-functions/commit/67e1858 [01:41:08] [02python-functions] 07github-actions[bot] 04deleted 03dependabot/pip/dot-github/mypy-1.16.1 at 03b0b116b 13https://api.github.com/repos/miraheze/python-functions/commit/b0b116b [01:45:21] miraheze/python-functions - github-merge-queue[bot] the build passed. [01:46:35] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f4589a8d3ade12c6610ae7c47daa124cc94ab9d3 [01:46:35] 02puppet/03main 07CosmicAlpha 03f4589a8 Disable monitoring for puppet-agent-timer… [01:51:17] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:51:22] PROBLEM - matomo151 Check unit status of matomo-optimize-last2 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:51:32] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:51:37] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:51:41] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:52:23] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is UNKNOWN: NRPE: Unable to read output [01:53:03] PROBLEM - matomo151 Check unit status of matomo-optimize-january on matomo151 is UNKNOWN: NRPE: Unable to read output [01:59:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/49a8bf66abe35ddd75255bf3eb2164064d2d0040 [01:59:04] 02puppet/03main 07CosmicAlpha 0349a8bf6 Don't ensure [02:00:13] RECOVERY - graylog161 Check unit status of prometheus_graylog_stats on graylog161 is OK: OK: Status of the systemd unit prometheus_graylog_stats [02:00:14] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-1 [02:00:18] RECOVERY - matomo151 Check unit status of matomo-optimize-january on matomo151 is OK: OK: Status of the systemd unit matomo-optimize-january [02:00:24] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [02:00:27] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [02:00:31] RECOVERY - matomo151 Check unit status of matomo-optimize-last2 on matomo151 is OK: OK: Status of the systemd unit matomo-optimize-last2 [02:00:38] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-2 [02:00:42] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [02:02:51] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics [02:04:11] PROBLEM - graylog161 Check unit status of prometheus_graylog_stats on graylog161 is CRITICAL: CRITICAL: Status of the systemd unit prometheus_graylog_stats [02:04:51] RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics [02:05:50] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-special-pages [02:25:54] [02pywikibot-config] 07Universal-Omega created 03timers (+1 new commit) 13https://github.com/miraheze/pywikibot-config/commit/ea3f590ce9de [02:25:55] 02pywikibot-config/03timers 07CosmicAlpha 03ea3f590 Add support for using systemd timers [02:25:59] [02pywikibot-config] 07Universal-Omega opened pull request #5: Add support for using systemd timers (03main...03timers) 13https://github.com/miraheze/pywikibot-config/pull/5 [02:26:04] [02pywikibot-config] 07coderabbitai[bot] commented on pull request #5:
[…] 13https://github.com/miraheze/pywikibot-config/pull/5#issuecomment-2978721442 [02:26:42] miraheze/pywikibot-config - Universal-Omega the build has errored. [02:26:56] [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03timers 13https://github.com/miraheze/pywikibot-config/commit/f2506237dd86e66e4599850aa0426a481ce98151 [02:26:56] 02pywikibot-config/03timers 07CosmicAlpha 03f250623 Add [02:27:34] miraheze/pywikibot-config - Universal-Omega the build has errored. [02:28:19] [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03timers 13https://github.com/miraheze/pywikibot-config/commit/c4e69b2ec08908267833c87d5b5bc7f16834a611 [02:28:19] 02pywikibot-config/03timers 07CosmicAlpha 03c4e69b2 Update [02:29:39] [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03timers 13https://github.com/miraheze/pywikibot-config/commit/f5583a4564232bafa9c4faa1058ff2bf4ca7816a [02:29:39] 02pywikibot-config/03timers 07CosmicAlpha 03f5583a4 Update [02:31:54] [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03timers 13https://github.com/miraheze/pywikibot-config/commit/a9f801d0864d6ed5475ad19d2ccd2e59ed4ec58d [02:31:54] 02pywikibot-config/03timers 07CosmicAlpha 03a9f801d Update [02:33:07] [02pywikibot-config] 07Universal-Omega merged pull request #5: Add support for using systemd timers (03main...03timers) 13https://github.com/miraheze/pywikibot-config/pull/5 [02:33:07] [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/pywikibot-config/commit/4af797b3fb1a4db578b4bc64b4499c5802307198 [02:33:07] 02pywikibot-config/03main 07CosmicAlpha 034af797b Add support for using systemd timers [02:33:10] [02pywikibot-config] 07Universal-Omega 04deleted 03timers at 03a9f801d 13https://api.github.com/repos/miraheze/pywikibot-config/commit/a9f801d [02:44:06] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5e49fa327f28981bb46704827615905c54be3666 [02:44:06] 02puppet/03main 07CosmicAlpha 035e49fa3 pywikibot: use systemd timers instead of crons [02:44:06] [02puppet] 07Universal-Omega merged pull request #4401: pywikibot: use systemd timers instead of crons (03main...03pywikibot-timer) 13https://github.com/miraheze/puppet/pull/4401 [02:44:07] [02puppet] 07Universal-Omega 04deleted 03pywikibot-timer at 038810653 13https://api.github.com/repos/miraheze/puppet/commit/8810653 [02:47:46] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:49:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ab39bb0faf98c306e2b885bac22738142a9d164f [02:49:42] 02puppet/03main 07CosmicAlpha 03ab39bb0 Fix [02:51:44] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [02:53:36] !log remove pywikibot crontab on bots171 (all crons on bots171 are migrated to systemd timers) [02:53:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:44] [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/pywikibot-config/commit/88dc8c1d1ca6f04f9f3299e99082af0fb16cec3e [02:54:44] 02pywikibot-config/03main 07CosmicAlpha 0388dc8c1 Remove old cron config [04:41:05] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats_lowlatency [04:45:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [04:57:43] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49869MiB (10% inode=99%); [05:11:44] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49916MiB (10% inode=99%); [06:27:05] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats_lowlatency [06:31:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [07:09:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750140560000&orgId=1&to=1750144193353 [07:34:53] PROBLEM - mwtask181 Check unit status of mediawiki_job_cleanup-upload-stash on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_cleanup-upload-stash [07:39:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750142300000&orgId=1&to=1750145900000 [08:01:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750143680000&orgId=1&to=1750147313355 [08:07:33] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8e8dd2af2810acbc830d8effa8f21747fbc26a66 [08:07:34] 02puppet/03main 07CosmicAlpha 038e8dd2a Update mattermost to 10.9.0 [08:16:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750144460000&orgId=1&to=1750148060000 [08:43:05] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats_lowlatency [08:47:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [09:45:24] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_update-special-pages [09:45:38] fun [11:25:28] [02puppet] 07RhinosF1 created 03RhinosF1-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/a5719cbe2975 [11:25:28] 02puppet/03RhinosF1-patch-1 07RhinosF1 03a5719cb mwdeploy: update gitinfo on --upgrade-{extensions,skins} [11:25:33] [02puppet] 07RhinosF1 opened pull request #4402: mwdeploy: update gitinfo on --upgrade-{extensions,skins} (03main...03RhinosF1-patch-1) 13https://github.com/miraheze/puppet/pull/4402 [11:25:39] [02puppet] 07coderabbitai[bot] commented on pull request #4402:
[…] 13https://github.com/miraheze/puppet/pull/4402#issuecomment-2979983148 [11:26:05] [02puppet] 07RhinosF1 self-assigned pull request #4402 (mwdeploy: update gitinfo on --upgrade-{extensions,skins}) 13https://github.com/miraheze/puppet/pull/4402 [11:26:08] [02puppet] 07RhinosF1 un-readied pull request #4402: mwdeploy: update gitinfo on --upgrade-{extensions,skins} (03main...03RhinosF1-patch-1) 13https://github.com/miraheze/puppet/pull/4402 [11:30:46] miraheze/puppet - RhinosF1 the build passed. [11:35:11] [02puppet] 07RhinosF1 commented on pull request #4402: @coderabbitai review 13https://github.com/miraheze/puppet/pull/4402#issuecomment-2980013228 [11:35:16] [02puppet] 07coderabbitai[bot] commented on pull request #4402:
[…] 13https://github.com/miraheze/puppet/pull/4402#issuecomment-2980013544 [12:15:05] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats_lowlatency [12:19:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [12:56:26] PROBLEM - cp171 NTP time on cp171 is WARNING: NTP WARNING: Offset -0.1030987203 secs [12:56:32] PROBLEM - cloud15 NTP time on cloud15 is WARNING: NTP WARNING: Offset -0.1026253104 secs [12:59:44] PROBLEM - mw161 NTP time on mw161 is WARNING: NTP WARNING: Offset -0.103743881 secs [12:59:44] PROBLEM - mem161 NTP time on mem161 is WARNING: NTP WARNING: Offset -0.1034070551 secs [12:59:44] PROBLEM - mwtask161 NTP time on mwtask161 is WARNING: NTP WARNING: Offset -0.1035731137 secs [12:59:44] PROBLEM - db161 NTP time on db161 is WARNING: NTP WARNING: Offset -0.1035948396 secs [12:59:44] PROBLEM - swiftobject161 NTP time on swiftobject161 is WARNING: NTP WARNING: Offset -0.1021378338 secs [12:59:45] PROBLEM - mw183 NTP time on mw183 is WARNING: NTP WARNING: Offset -0.1027916968 secs [12:59:45] PROBLEM - reports171 NTP time on reports171 is WARNING: NTP WARNING: Offset -0.1038715839 secs [12:59:46] PROBLEM - mw192 NTP time on mw192 is WARNING: NTP WARNING: Offset -0.1032581329 secs [12:59:46] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089 [12:59:47] PROBLEM - mwtask171 NTP time on mwtask171 is WARNING: NTP WARNING: Offset -0.1027038395 secs [12:59:47] PROBLEM - mwtask181 NTP time on mwtask181 is WARNING: NTP WARNING: Offset -0.1022814512 secs [12:59:48] PROBLEM - os151 NTP time on os151 is WARNING: NTP WARNING: Offset -0.1021715105 secs [12:59:48] PROBLEM - cp191 NTP time on cp191 is WARNING: NTP WARNING: Offset -0.1028057933 secs [12:59:49] PROBLEM - mw181 NTP time on mw181 is WARNING: NTP WARNING: Offset -0.1028859615 secs [13:00:04] PROBLEM - worldless.wiki - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:04] PROBLEM - os161 NTP time on os161 is WARNING: NTP WARNING: Offset -0.1036871672 secs [13:00:04] PROBLEM - os162 NTP time on os162 is WARNING: NTP WARNING: Offset -0.103824228 secs [13:00:04] PROBLEM - eventgate181 NTP time on eventgate181 is WARNING: NTP WARNING: Offset -0.1037213405 secs [13:00:04] PROBLEM - rdb151 NTP time on rdb151 is WARNING: NTP WARNING: Offset -0.1035548747 secs [13:17:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:26:34] RECOVERY - az.sanarsiv.org - Cloudflare on sslhost is OK: OK - Certificate 'az.sanarsiv.org' will expire on Sat 06 Sep 2025 02:27:04 PM GMT +0000. [13:27:08] RECOVERY - worldless.wiki - Cloudflare on sslhost is OK: OK - Certificate 'worldless.wiki' will expire on Fri 29 Aug 2025 05:15:09 PM GMT +0000. [13:27:42] RECOVERY - rainverse.wiki - Cloudflare on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Mon 11 Aug 2025 04:22:36 AM GMT +0000. [13:28:28] RECOVERY - worldtriggerwiki.com - Cloudflare on sslhost is OK: OK - Certificate 'worldtriggerwiki.com' will expire on Wed 13 Aug 2025 06:54:22 PM GMT +0000. [13:28:44] RECOVERY - pizzatower.wiki - Cloudflare on sslhost is OK: OK - Certificate 'pizzatower.wiki' will expire on Wed 20 Aug 2025 11:40:27 PM GMT +0000. [13:29:00] RECOVERY - darkwaterswiki.com - Cloudflare on sslhost is OK: OK - Certificate 'darkwaterswiki.com' will expire on Fri 15 Aug 2025 04:37:27 AM GMT +0000. [13:29:22] RECOVERY - portalsofphereon.com - Cloudflare on sslhost is OK: OK - Certificate 'portalsofphereon.com' will expire on Sat 09 Aug 2025 01:50:03 AM GMT +0000. [13:30:13] RECOVERY - svewiki.com - Cloudflare on sslhost is OK: OK - Certificate 'svewiki.com' will expire on Mon 21 Jul 2025 03:20:09 AM GMT +0000. [13:30:27] RECOVERY - www.backroomszh.wiki - Cloudflare on sslhost is OK: OK - Certificate 'www.backroomszh.wiki' will expire on Thu 04 Sep 2025 05:14:31 AM GMT +0000. [13:30:42] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:31:05] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats_lowlatency [14:35:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [16:17:05] RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats_lowlatency [16:21:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats_lowlatency [17:00:29] [02puppet] 07Universal-Omega created 03absent-swift_dispersion_stats_lowlatency (+1 new commit) 13https://github.com/miraheze/puppet/commit/8dcd8ddb39f6 [17:00:29] 02puppet/03absent-swift_dispersion_stats_lowlatency 07CosmicAlpha 038dcd8dd absent unused systemd timer swift_dispersion_stats_lowlatency [17:00:34] [02puppet] 07Universal-Omega opened pull request #4403: absent unused systemd timer swift_dispersion_stats_lowlatency (03main...03absent-swift_dispersion_stats_lowlatency) 13https://github.com/miraheze/puppet/pull/4403 [17:00:40] [02puppet] 07coderabbitai[bot] commented on pull request #4403:
[…] 13https://github.com/miraheze/puppet/pull/4403#issuecomment-2981131713 [17:14:22] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'ThemeToggle'} to test151 [17:14:24] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'ThemeToggle'} to test151 - SUCCESS in 2s [17:38:11] [02puppet] 07Universal-Omega created 03systemd-monitoring_docs_url (+1 new commit) 13https://github.com/miraheze/puppet/commit/6a3b8edc18e5 [17:38:11] 02puppet/03systemd-monitoring_docs_url 07CosmicAlpha 036a3b8ed systemd: add support for adding docs url to monitoring [17:38:17] [02puppet] 07Universal-Omega opened pull request #4404: systemd: add support for adding docs url to monitoring (03main...03systemd-monitoring_docs_url) 13https://github.com/miraheze/puppet/pull/4404 [17:38:23] [02puppet] 07coderabbitai[bot] commented on pull request #4404:
[…] 13https://github.com/miraheze/puppet/pull/4404#issuecomment-2981236506 [17:41:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitoring_docs_url 13https://github.com/miraheze/puppet/commit/bb13f3be56229067feb845b5224baed6451922ef [17:41:36] 02puppet/03systemd-monitoring_docs_url 07CosmicAlpha 03bb13f3b Add [17:53:11] [02puppet] 07Universal-Omega merged pull request #4404: systemd: add support for adding docs url to monitoring (03main...03systemd-monitoring_docs_url) 13https://github.com/miraheze/puppet/pull/4404 [17:53:11] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/43ca605b5bda3a7a2ae4815203498c71a3b7f57f [17:53:11] 02puppet/03main 07CosmicAlpha 0343ca605 systemd: add support for adding docs url to monitoring [17:53:13] [02puppet] 07Universal-Omega 04deleted 03systemd-monitoring_docs_url at 03bb13f3b 13https://api.github.com/repos/miraheze/puppet/commit/bb13f3b [17:53:56] [02puppet] 07Universal-Omega merged pull request #4403: absent unused systemd timer swift_dispersion_stats_lowlatency (03main...03absent-swift_dispersion_stats_lowlatency) 13https://github.com/miraheze/puppet/pull/4403 [17:53:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/391953fc947d4d9f1b49c26f14605d7a67e5b017 [17:53:56] 02puppet/03main 07CosmicAlpha 03391953f absent unused systemd timer swift_dispersion_stats_lowlatency [17:53:57] [02puppet] 07Universal-Omega 04deleted 03absent-swift_dispersion_stats_lowlatency at 038dcd8dd 13https://api.github.com/repos/miraheze/puppet/commit/8dcd8dd [17:59:05] PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats_lowlatency on swiftproxy171 is UNKNOWN: NRPE: Command 'check_Check unit status of swift_dispersion_stats_lowlatency' not defined [18:01:56] [02puppet] 07Universal-Omega created 03swift-timer-remove (+1 new commit) 13https://github.com/miraheze/puppet/commit/ee98faa1dbcb [18:01:56] 02puppet/03swift-timer-remove 07CosmicAlpha 03ee98faa swift: remove absented timer [18:02:01] [02puppet] 07Universal-Omega opened pull request #4405: swift: remove absented timer (03main...03swift-timer-remove) 13https://github.com/miraheze/puppet/pull/4405 [18:02:06] [02puppet] 07coderabbitai[bot] commented on pull request #4405:
[…] 13https://github.com/miraheze/puppet/pull/4405#issuecomment-2981321687 [18:02:53] [02puppet] 07Universal-Omega merged pull request #4405: swift: remove absented timer (03main...03swift-timer-remove) 13https://github.com/miraheze/puppet/pull/4405 [18:02:54] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/dab2474f343eafe36084decd5a4a5bde0e7fde6d [18:02:54] 02puppet/03main 07CosmicAlpha 03dab2474 swift: remove absented timer [18:02:55] [02puppet] 07Universal-Omega 04deleted 03swift-timer-remove at 03ee98faa 13https://api.github.com/repos/miraheze/puppet/commit/ee98faa [18:41:32] [02puppet] 07Universal-Omega created 03syslog (+1 new commit) 13https://github.com/miraheze/puppet/commit/4933753c71e3 [18:41:32] 02puppet/03syslog 07CosmicAlpha 034933753 Fix syslog when using systemd-timer-mail-wrapper [18:41:36] [02puppet] 07Universal-Omega opened pull request #4406: Fix syslog when using systemd-timer-mail-wrapper (03main...03syslog) 13https://github.com/miraheze/puppet/pull/4406 [18:41:42] [02puppet] 07coderabbitai[bot] commented on pull request #4406:
[…] 13https://github.com/miraheze/puppet/pull/4406#issuecomment-2981459073 [18:46:41] [02puppet] 07Universal-Omega closed pull request #4406: Fix syslog when using systemd-timer-mail-wrapper (03main...03syslog) 13https://github.com/miraheze/puppet/pull/4406 [18:47:54] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/14d38b3153bafef7c314fb1dcb39e3d0f11eea55 [18:47:54] 02puppet/03main 07CosmicAlpha 0314d38b3 Update defaults [18:49:23] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e05ee09a85428a1bef0cd05972d4961fd00da045 [18:49:24] 02puppet/03main 07CosmicAlpha 03e05ee09 Don't send mail [18:55:54] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-special-pages [18:57:19] RECOVERY - mwtask181 Check unit status of mediawiki_job_cleanup-upload-stash on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash [19:01:46] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f53fe7479a1b4a1999dfe9d7f66d22dfa7d086dc [19:01:46] 02puppet/03main 07CosmicAlpha 03f53fe74 graylog: absent unused timer [19:03:40] PROBLEM - graylog161 Check unit status of prometheus_graylog_stats on graylog161 is UNKNOWN: NRPE: Command 'check_Check unit status of prometheus_graylog_stats' not defined [19:07:41] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1560dc24b9a6854b1affcbba477fa9bab01bad07 [19:07:41] 02puppet/03main 07CosmicAlpha 031560dc2 graylog: remove unused exporter [19:38:11] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43] [19:47:40] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png] [19:48:43] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.43] [19:57:36] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:04:02] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:14:21] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:58:29] [02puppet] 07Universal-Omega created 03host-v4-v6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/8965aaae99ba [20:58:29] 02puppet/03host-v4-v6 07CosmicAlpha 038965aaa monitoring::hosts: always include both IPv4 and IPv6 addresses [20:58:32] [02puppet] 07Universal-Omega opened pull request #4407: monitoring::hosts: always include both IPv4 and IPv6 addresses (03main...03host-v4-v6) 13https://github.com/miraheze/puppet/pull/4407 [20:58:37] [02puppet] 07coderabbitai[bot] commented on pull request #4407:
[…] 13https://github.com/miraheze/puppet/pull/4407#issuecomment-2981814402 [21:03:22] [02puppet] 07Universal-Omega closed pull request #4392: monitoring::hosts: always use address not address6 (03main...03host-addr) 13https://github.com/miraheze/puppet/pull/4392 [21:03:25] [02puppet] 07Universal-Omega 04deleted 03host-addr at 03505238f 13https://api.github.com/repos/miraheze/puppet/commit/505238f [21:16:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/09086b5cb3322fa6d062eda40c4cb5afd7c00071 [21:16:36] 02puppet/03main 07CosmicAlpha 0309086b5 Update mattermost to 10.9.1 [22:40:00] [02mw-config] 07SomeMWDev opened pull request #5988: Add wgNortonSiteVerificationKey to ManageWiki (07miraheze:03main...07SomeMWDev:03norton-site-key-setting) 13https://github.com/miraheze/mw-config/pull/5988 [22:40:52] miraheze/mw-config - SomeMWDev the build passed. [22:41:52] [02mw-config] 07SomeMWDev merged pull request #5988: Add wgNortonSiteVerificationKey to ManageWiki (07miraheze:03main...07SomeMWDev:03norton-site-key-setting) 13https://github.com/miraheze/mw-config/pull/5988 [22:41:53] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/7a83b4ffe26f4cf899432f659aacc18e028799c8 [22:41:53] 02mw-config/03main 07SomeRandomDeveloper 037a83b4f Add wgNortonSiteVerificationKey to ManageWiki (#5988)… [22:42:09] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [22:42:10] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [22:42:42] miraheze/mw-config - SomeMWDev the build passed. [22:42:44] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:43:10] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [22:44:07] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03message-updater 13https://github.com/miraheze/ManageWiki/commit/b2fe9568383941e6e50dacc9d2f08ab162a70daa [22:44:07] 02ManageWiki/03message-updater 07CosmicAlpha 03b2fe956 Merge branch 'main' into message-updater [22:47:42] miraheze/ManageWiki - Universal-Omega the build has errored. [22:48:04] [02mw-config] 07SomeMWDev opened pull request #5989: Remove dysfunctional settings (07miraheze:03main...07SomeMWDev:03remove-removed-settings) 13https://github.com/miraheze/mw-config/pull/5989 [22:48:55] miraheze/mw-config - SomeMWDev the build passed. [22:51:27] [02mw-config] 07SomeMWDev merged pull request #5989: Remove dysfunctional settings (07miraheze:03main...07SomeMWDev:03remove-removed-settings) 13https://github.com/miraheze/mw-config/pull/5989 [22:51:29] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/0c138d1b950d31ec3d8daee3d5cb7d64f1b28fa3 [22:51:30] 02mw-config/03main 07SomeRandomDeveloper 030c138d1 Remove dysfunctional settings (#5989) [22:51:39] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [22:51:40] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [22:52:07] [02puppet] 07Universal-Omega created 03remove-he-ipv6 from 03main (+0 new commit) 13https://github.com/miraheze/puppet/compare/remove-he-ipv6 [22:52:22] miraheze/mw-config - SomeMWDev the build passed. [22:52:54] !log [somerandomdeveloper@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateWikiSettings --setting=wgAlexaSiteVerificationKey --remove (START) [22:52:55] [02puppet] 07Universal-Omega pushed 1 new commit to 03remove-he-ipv6 13https://github.com/miraheze/puppet/commit/6ef6d68203157f50b7346cb66d9d173ad8fa82da [22:52:55] 02puppet/03remove-he-ipv6 07CosmicAlpha 036ef6d68 Remove he-ipv6 checks… [22:53:02] !log [somerandomdeveloper@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateWikiSettings --setting=wgAlexaSiteVerificationKey --remove (END - exit=0) [22:53:12] [02puppet] 07Universal-Omega opened pull request #4408: Remove he-ipv6 checks (03main...03remove-he-ipv6) 13https://github.com/miraheze/puppet/pull/4408 [22:53:17] [02puppet] 07coderabbitai[bot] commented on pull request #4408:
[…] 13https://github.com/miraheze/puppet/pull/4408#issuecomment-2982052713 [22:53:51] !log [somerandomdeveloper@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateWikiSettings --setting=wgTabberNeueUseCodex --remove (START) [22:53:59] !log [somerandomdeveloper@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:PopulateWikiSettings --setting=wgTabberNeueUseCodex --remove (END - exit=0) [22:55:17] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:55:43] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [22:56:15] !log [somerandomdeveloper@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php ManageWiki:PopulateWikiSettings --setting=wgAlexaSiteVerificationKey --remove (START) [22:59:28] [02puppet] 07Universal-Omega merged pull request #4408: Remove he-ipv6 checks (03main...03remove-he-ipv6) 13https://github.com/miraheze/puppet/pull/4408 [22:59:28] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e868ffc9df50f3d2306d936448b61e4b9454b043 [22:59:28] 02puppet/03main 07CosmicAlpha 03e868ffc Remove he-ipv6 checks… [22:59:29] [02puppet] 07Universal-Omega 04deleted 03remove-he-ipv6 at 036ef6d68 13https://api.github.com/repos/miraheze/puppet/commit/6ef6d68 [23:03:22] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ed973197a881e5ab2e1d720bcd9080ba3a5841f3 [23:03:22] 02puppet/03main 07CosmicAlpha 03ed97319 Remove unused monitoring::wiki [23:08:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03techdocs-automation 13https://github.com/miraheze/puppet/commit/f843f3fe5d4ad3293a941ec8ca58bec81793a936 [23:08:07] 02puppet/03techdocs-automation 07CosmicAlpha 03f843f3f Use fsslc.wtnet [23:08:20] miraheze/puppet - Universal-Omega the build passed. [23:08:54] miraheze/puppet - Universal-Omega the build passed. [23:09:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03techdocs-automation 13https://github.com/miraheze/puppet/commit/cf60e08880f1c7a65ee6bdafca6a044e1515410f [23:09:04] 02puppet/03techdocs-automation 07CosmicAlpha 03cf60e08 Use fsslc.wtnet [23:09:21] miraheze/puppet - Universal-Omega the build passed. [23:10:20] miraheze/puppet - Universal-Omega the build passed. [23:17:27] [02puppet] 07Universal-Omega pushed 1 new commit to 03techdocs-automation 13https://github.com/miraheze/puppet/commit/9dd6edc607224be9e5144762456224b5994e46ba [23:17:27] 02puppet/03techdocs-automation 07CosmicAlpha 039dd6edc Use mediawiki::periodic_job [23:19:24] miraheze/puppet - Universal-Omega the build passed. [23:24:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03techdocs-automation 13https://github.com/miraheze/puppet/commit/d95223e489005e950f4e315615b0ff7374310c64 [23:24:47] 02puppet/03techdocs-automation 07CosmicAlpha 03d95223e Revert [23:26:01] miraheze/puppet - Universal-Omega the build passed. [23:29:23] [02puppet] 07Universal-Omega pushed 1 new commit to 03techdocs-automation 13https://github.com/miraheze/puppet/commit/c5b09ab96e635ea62990b4756956f859dd50ad1f [23:29:23] 02puppet/03techdocs-automation 07CosmicAlpha 03c5b09ab Add ensure [23:30:32] miraheze/puppet - Universal-Omega the build passed.