[00:02:59] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.75, 18.67, 13.89 [00:06:51] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.38, 19.75, 15.41 [01:19:36] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:25:36] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:41:37] PROBLEM - jobrunner2 APT on jobrunner2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [03:06:29] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [03:39:54] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUsBF [03:39:55] [02miraheze/mediawiki] 07paladox 0388a5599 - Update Cosmos [04:03:32] !log rebuild lc on mw* [04:03:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:36] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:21:11] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:21:24] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:25:12] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:25:22] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:25:36] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUs08 [04:37:57] [02miraheze/mediawiki] 07paladox 030bc00eb - Update Cosmos [04:38:29] !log rebuild lc on jobrunner* and test2 [04:38:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:29] RECOVERY - mw5 APT on mw5 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:09:16] [02mw-config] 07RhinosF1 closed pull request 03#3230: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUsse [06:09:30] [02ManageWiki] 07RhinosF1 commented on pull request 03#182: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUsu1 [06:09:32] [02ManageWiki] 07RhinosF1 closed pull request 03#182: fix project name to "acl*security" from "Security" in README.md - 13https://git.io/JUssC [06:23:37] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:25:25] [02mediawiki] 07Reception123 closed pull request 03#148: Add RightFunctions extension (T6140) - 13https://git.io/JUsJJ [06:25:27] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/JUsz0 [06:25:28] [02miraheze/mediawiki] 07Universal-Omega 03bb87a6e - Add RightFunctions extension (T6140) (#148) * Update .gitmodules * Update .gitmodules * Add Rightfunctions extension (T6140) [06:31:40] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:32:39] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [06:40:27] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.013 second response time [06:43:41] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:44:16] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 1241 bytes in 0.041 second response time [06:49:40] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:57:41] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:00:57] [02mw-config] 07Reception123 commented on pull request 03#3229: Add RightFunctions extension (T6140) - 13https://git.io/JUs2T [07:07:23] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [07:20:26] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUs21 [07:20:28] [02miraheze/ssl] 07Reception123 03be97f36 - www for shkilev.online [07:33:05] PROBLEM - www.shkilev.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.shkilev.online could not be found [07:58:15] [02CreateWiki] 07RhinosF1 commented on pull request 03#158: Added Phabricator and Discord Links to default mainpage; changed external link style to interwiki format - 13https://git.io/JUsaX [07:58:31] [02CreateWiki] 07RhinosF1 edited a comment on pull request 03#158: Added Phabricator and Discord Links to default mainpage; changed external link style to interwiki format - 13https://git.io/JUsaX [08:44:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.36, 17.95, 15.15 [08:46:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.35, 17.11, 15.20 [11:44:20] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [13:07:29] [02mw-config] 07Universal-Omega synchronize pull request 03#3229: Add RightFunctions extension (T6140) - 13https://git.io/JU3zq [13:10:10] [02mw-config] 07Universal-Omega synchronize pull request 03#3229: Add RightFunctions extension (T6140) - 13https://git.io/JU3zq [13:17:17] [02mw-config] 07Reception123 closed pull request 03#3229: Add RightFunctions extension (T6140) - 13https://git.io/JU3zq [13:17:19] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JUsSB [13:17:20] [02miraheze/mw-config] 07Universal-Omega 03232f687 - Add RightFunctions extension (T6140) (#3229) * Add RightFunctions extension (T6140) * Add RightFunctions extension (T6140) * Add RightFunctions extension (T6140) * Add RightFunctions extension (T6140) * Update to RightFunctions.php * require_once for REL1_34 version [13:23:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.37, 20.35, 18.78 [13:23:36] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr* [13:23:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:55] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [13:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:58] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.46, 23.29, 20.02 [13:28:53] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.93, 23.16, 20.86 [13:32:48] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 34.96, 26.10, 22.25 [13:40:54] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:41:01] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:41:10] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:41:46] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:45:46] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:46:51] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:46:58] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:47:09] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:48:32] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.32, 21.97, 23.15 [13:52:32] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 63.82, 36.55, 27.93 [13:54:11] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 10.33, 12.90, 6.81 [13:55:15] PROBLEM - private.revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for private.revi.wiki could not be found [13:58:07] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.84, 6.90, 5.70 [14:00:04] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 0.91, 4.92, 5.11 [14:00:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUsHI [14:00:21] [02miraheze/services] 07MirahezeSSLBot 037eee8a2 - BOT: Updating services config for wikis [14:00:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.35, 21.68, 23.85 [14:02:11] RECOVERY - private.revi.wiki - reverse DNS on sslhost is OK: rDNS OK - private.revi.wiki reverse DNS resolves to cp6.miraheze.org [14:02:23] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.67, 22.16, 23.73 [14:04:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.51, 19.47, 22.55 [14:07:03] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.permanentfuturelab.wiki could not be found [14:08:22] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [14:10:16] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.29, 22.45, 22.80 [14:12:15] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for permanentfuturelab.wiki could not be found [14:13:45] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - www.permanentfuturelab.wiki reverse DNS resolves to cp6.miraheze.org [14:14:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.13, 22.43, 22.88 [14:19:12] RECOVERY - permanentfuturelab.wiki - reverse DNS on sslhost is OK: rDNS OK - permanentfuturelab.wiki reverse DNS resolves to cp6.miraheze.org [14:28:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.71, 16.26, 19.59 [14:35:26] [02mediawiki] 07Universal-Omega opened pull request 03#149: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs7I [14:37:00] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.37, 23.38, 21.08 [14:38:57] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.84, 20.35, 20.24 [14:42:51] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.68, 21.01, 20.54 [14:44:49] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.41, 18.84, 19.82 [14:46:11] [02mw-config] 07Universal-Omega opened pull request 03#3231: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs7i [14:48:35] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [14:51:51] [02mediawiki] 07Reception123 closed pull request 03#149: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs7I [14:51:52] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/JUs7F [14:51:54] [02miraheze/mediawiki] 07Universal-Omega 03fdb8762 - Add DeleteUserPages extension (T6139) (#149) [14:53:46] [02mediawiki] 07Universal-Omega commented on pull request 03#149: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs7p [14:55:16] [02mediawiki] 07Reception123 commented on pull request 03#149: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs5v [14:56:07] [02mediawiki] 07Universal-Omega commented on pull request 03#149: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs5J [15:02:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.44, 19.34, 19.42 [15:03:16] [02mediawiki] 07Universal-Omega opened pull request 03#150: add ignore = dirty where missing - 13https://git.io/JUs54 [15:04:27] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.72, 18.82, 19.19 [15:06:49] [02mediawiki] 07Reception123 closed pull request 03#150: add ignore = dirty where missing - 13https://git.io/JUs54 [15:06:51] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUs5V [15:06:52] [02miraheze/mediawiki] 07Universal-Omega 0349067d6 - add ignore = dirty where missing (#150) [15:17:32] [02mw-config] 07Reception123 closed pull request 03#3231: Add DeleteUserPages extension (T6139) - 13https://git.io/JUs7i [15:17:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JUsdl [15:17:35] [02miraheze/mw-config] 07Universal-Omega 031011616 - Add DeleteUserPages extension (T6139) (#3231) * Add DeleteUserPages extension (T6139) * Add DeleteUserPages extension (T6139) * Add DeleteUserPages extension (T6139) * Add DeleteUserPages extension (T6139) [15:23:47] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr* [15:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:57] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.88, 21.25, 20.03 [15:24:04] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [15:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:54] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.73, 23.37, 20.94 [15:27:52] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.99, 23.16, 21.18 [15:29:49] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.35, 26.82, 22.73 [15:29:50] PROBLEM - bacula2 Bacula Static 1 on bacula2 is WARNING: WARNING: Full, 11404431 files, 584.4GB, 2020-08-21 15:26:00 (2.1 weeks ago) [15:32:23] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.56, 6.76, 4.53 [15:32:56] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.83, 6.42, 5.06 [15:34:22] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.95, 6.50, 4.73 [15:36:57] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.71, 6.64, 5.48 [15:38:46] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 9.13, 7.35, 5.83 [15:40:53] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.76, 7.32, 6.02 [15:41:21] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:41:24] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:41:31] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.35, 6.67, 4.56 [15:42:24] [02mw-config] 07Universal-Omega opened pull request 03#3232: Add permissions by default to the 'user' user group for Extension:Del… - 13https://git.io/JUsbk [15:42:50] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 9.98, 8.13, 6.46 [15:43:30] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.52, 6.10, 4.61 [15:44:36] [02mw-config] 07Reception123 closed pull request 03#3232: Add permissions by default to the 'user' user group for Extension:Del… - 13https://git.io/JUsbk [15:44:38] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JUsbc [15:44:40] [02miraheze/mw-config] 07Universal-Omega 03b721efc - Add permissions by default to the 'user' user group for Extension:DeleteUserPages (T6139) (#3232) [15:46:50] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 6.00, 7.61, 6.76 [15:46:57] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 6.58, 7.61, 6.67 [15:47:21] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [15:47:32] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.74, 7.67, 5.53 [15:49:28] PROBLEM - wiki.dom64.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dom64.xyz could not be found [15:50:47] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.05, 6.24, 6.40 [15:52:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb [15:52:47] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.42, 6.70, 6.61 [15:53:30] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 4.67, 7.18, 6.11 [15:54:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:55:13] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.019 second response time [15:55:29] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 3.14, 5.81, 5.73 [15:56:22] RECOVERY - wiki.dom64.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.dom64.xyz reverse DNS resolves to cp6.miraheze.org [15:57:12] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:59:22] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 1241 bytes in 0.054 second response time [15:59:39] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.96, 8.06, 6.67 [16:00:20] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 195s [16:00:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [16:00:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [16:01:42] PROBLEM - infectowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for infectowiki.com could not be found [16:01:46] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.28, 8.00, 6.85 [16:02:18] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [16:02:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:02:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:05:54] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 9.15, 8.16, 7.13 [16:06:46] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [16:08:38] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: rDNS OK - infectowiki.com reverse DNS resolves to cp7.miraheze.org [16:09:53] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.34, 7.79, 7.24 [16:11:53] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 2.46, 5.75, 6.56 [16:18:46] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [16:21:37] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildmessages.php --wiki loginwiki on mw* and jbr* [16:21:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:10] !log cosmoswiki not loginwiki for previous [16:22:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:39] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.36, 16.86, 23.43 [16:37:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.99, 20.71, 21.73 [16:39:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.09, 17.88, 20.57 [16:41:01] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [16:41:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.07, 17.33, 20.05 [17:12:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.59, 22.20, 20.22 [17:16:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.85, 22.71, 20.82 [17:18:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.12, 20.38, 20.20 [19:29:07] RECOVERY - mw4 APT on mw4 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:29:18] !log installing security fixes to resolve icinga alerts [19:29:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:55] RECOVERY - mw7 APT on mw7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:30:21] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:30:33] RECOVERY - mw6 APT on mw6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:30:59] RECOVERY - test2 APT on test2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:32:20] RECOVERY - services1 APT on services1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:32:21] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JUGfp [19:32:23] [02miraheze/mediawiki] 07Reception123 03e14fb5c - Update Cosmos skin [19:35:44] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.20, 21.15, 18.73 [19:37:41] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.19, 18.69, 18.10 [19:43:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.48, 21.30, 19.26 [19:44:13] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d] [19:47:41] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.01, 22.96, 20.30 [19:51:13] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:51:14] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:51:54] SPF|Cloud: you around for a quick discord message? [19:51:57] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:52:05] in a few mins [19:52:09] Kk [19:53:42] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.06, 23.97, 21.92 [19:55:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.00, 23.42, 21.86 [19:57:11] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:57:12] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:57:39] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.53, 22.67, 21.78 [20:05:31] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.15, 24.54, 22.81 [20:11:10] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:11:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.04, 23.64, 23.47 [20:13:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 23.92, 24.29, 23.74 [20:17:14] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.57, 21.70, 22.93 [20:19:09] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:25:02] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.56, 20.73, 21.52 [20:27:45] PROBLEM - orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:29:02] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'miraheze.com' expires in 7 day(s) (Sun 13 Sep 2020 20:28:55 GMT +0000). [20:30:54] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.30, 23.01, 22.61 [20:31:02] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:32:46] PROBLEM - meta.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:33:36] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Sun 13 Sep 2020 20:27:17 GMT +0000). [20:46:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.49, 19.15, 20.34 [21:25:01] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.50, 22.37, 19.93 [21:29:00] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.16, 22.27, 20.57 [21:32:58] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.47, 19.35, 19.89 [22:10:10] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 38.01, 23.55, 19.63 [22:11:08] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 11.18, 8.55, 4.93 [22:12:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.99, 21.97, 19.56 [22:13:05] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.57, 6.20, 4.49 [22:14:09] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.31, 19.25, 18.88 [22:26:11] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.11, 25.34, 21.21 [22:28:10] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.93, 22.88, 20.92 [22:30:11] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.73, 19.74, 19.99 [23:03:10] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 7.07, 3.55, 1.48 [23:09:02] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.00, 1.07, 1.68