[00:03:46] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 22.86, 24.86, 20.11 [00:04:39] [02mw-config] 07Universal-Omega opened pull request 03#3428: Remove DeleteUserPages extension - 13https://git.io/JkmnE [00:05:17] [02mw-config] 07Universal-Omega synchronize pull request 03#3428: Remove DeleteUserPages extension - 13https://git.io/JkmnE [00:05:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.06, 23.61, 20.20 [00:07:36] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.16, 19.52, 19.08 [00:11:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.15, 21.59, 20.11 [00:14:55] [02mw-config] 07Universal-Omega commented on pull request 03#3428: Remove DeleteUserPages extension - 13https://git.io/Jkmce [00:14:57] [02mw-config] 07Universal-Omega closed pull request 03#3428: Remove DeleteUserPages extension - 13https://git.io/JkmnE [00:15:19] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.83, 20.00, 19.91 [00:44:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.67, 22.18, 18.88 [00:46:13] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.33, 19.34, 18.25 [00:55:34] miraheze/mw-config/master/31454b7 - Universal Omega The build passed. https://travis-ci.org/miraheze/mw-config/builds/743293436 [01:00:06] [02mw-config] 07Universal-Omega opened pull request 03#3429: Remove config that does not even exist - 13https://git.io/JkmWP [01:00:36] [02mw-config] 07Universal-Omega synchronize pull request 03#3429: Remove config that does not even exist - 13https://git.io/JkmWP [01:01:33] [02mw-config] 07Universal-Omega reviewed pull request 03#3426 commit - 13https://git.io/JkmWS [01:03:34] [02mw-config] 07paladox closed pull request 03#3429: Remove config that does not even exist - 13https://git.io/JkmWP [01:03:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkmWH [01:03:37] [02miraheze/mw-config] 07Universal-Omega 034a56047 - Remove config that does not even exist (#3429) * Remove config that does not even exist * spacing [01:18:49] [02mw-config] 07Universal-Omega opened pull request 03#3430: Remove Test2 override for extension set in MWE - 13https://git.io/Jkmlb [02:34:29] [02mw-config] 07soratako synchronize pull request 03#3426: Add some pages to allowlist (whitelist) for swiki - 13https://git.io/Jkmqn [03:43:56] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:44:18] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 6.59, 5.79, 2.76 [03:45:56] RECOVERY - ns1 APT on ns1 is OK: APT OK: 6 packages available for upgrade (0 critical updates). [03:48:17] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.00, 1.17, 1.65 [06:42:46] [02mw-config] 07Universal-Omega reviewed pull request 03#3426 commit - 13https://git.io/JkmHW [06:52:20] [02mw-config] 07Reception123 closed pull request 03#3430: Remove Test2 override for extension set in MWE - 13https://git.io/Jkmlb [06:52:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkmQ6 [06:52:23] [02miraheze/mw-config] 07Universal-Omega 03c0e7c05 - Remove Test2 override for extension set in MWE (#3430) [06:53:35] [02mw-config] 07Reception123 commented on pull request 03#3424: Remove Extension:WhoIsWatching - 13https://git.io/JkmQd [06:53:52] [02mw-config] 07Reception123 closed pull request 03#3424: Remove Extension:WhoIsWatching - 13https://git.io/JkmkI [06:53:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JkmQN [06:53:55] [02miraheze/mw-config] 07Universal-Omega 033833fa5 - Remove Extension:WhoIsWatching (#3424) * Remove Extension:WhoIsWatching * Remove * remove [07:23:14] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.31, 15.84, 11.77 [07:25:14] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 11.03, 13.67, 11.47 [07:35:40] [02mw-config] 07Universal-Omega opened pull request 03#3431: remove wiki overrides for MWS setting - 13https://git.io/Jkmb6 [07:42:38] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.65, 19.87, 17.85 [07:44:33] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.53, 19.36, 17.96 [07:50:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.44, 20.68, 19.03 [07:54:12] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.00, 19.14, 18.99 [07:57:19] [02mw-config] 07Universal-Omega synchronize pull request 03#3421: Migrate $wgCompressRevisions to managewiki-restricted - 13https://git.io/Jkmfn [08:02:31] [02mw-config] 07Reception123 closed pull request 03#3431: remove wiki overrides for MWS setting - 13https://git.io/Jkmb6 [08:02:32] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkmxL [08:02:34] [02miraheze/mw-config] 07Universal-Omega 03bf90408 - remove wiki overrides for MWS setting (#3431) [09:12:11] [02mw-config] 07Universal-Omega opened pull request 03#3432: Migrate $wgCategoryPagingLimit to ManageWikiSettings - 13https://git.io/JkYJG [09:15:10] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [09:22:16] [02mw-config] 07Universal-Omega synchronize pull request 03#3432: Migrate $wgCategoryPagingLimit to ManageWikiSettings - 13https://git.io/JkYJG [09:23:10] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:17:45] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 19.43, 9.52, 4.70 [10:18:21] PROBLEM - intp.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for intp.miraheze.org could not be found [10:19:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.61, 28.43, 21.49 [10:19:44] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.43, 6.86, 4.33 [10:21:44] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.69, 5.44, 4.09 [10:23:02] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.08, 23.65, 21.07 [10:24:58] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.36, 19.69, 19.92 [10:25:05] RECOVERY - intp.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - intp.miraheze.org reverse DNS resolves to cp6.miraheze.org [11:20:06] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mastodon.kr' expires in 15 day(s) (Sun 29 Nov 2020 11:14:35 GMT +0000). [11:27:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkYsA [11:27:53] [02miraheze/ssl] 07MirahezeSSLBot 03e1814eb - Bot: Update SSL cert for wiki.mastodon.kr [11:33:55] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Thu 11 Feb 2021 10:27:45 GMT +0000. [12:41:44] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 14.16, 8.36, 4.16 [12:42:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.06, 26.38, 20.10 [12:43:44] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 3.10, 6.01, 3.81 [12:44:02] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.86, 22.75, 19.56 [12:46:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.14, 19.40, 18.72 [13:04:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.51, 20.53, 19.47 [13:06:23] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.55, 20.37, 19.54 [13:39:01] PROBLEM - shkilev.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'shkilev.online' expires in 15 day(s) (Sun 29 Nov 2020 13:33:59 GMT +0000). [13:42:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkYau [13:42:05] [02miraheze/ssl] 07MirahezeSSLBot 03a753a60 - Bot: Update SSL cert for shkilev.online [13:43:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.10, 20.41, 18.62 [13:47:15] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.89, 18.88, 18.48 [13:51:06] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.53, 22.64, 20.07 [13:54:56] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.09, 20.15, 19.90 [13:59:38] RECOVERY - shkilev.online - LetsEncrypt on sslhost is OK: OK - Certificate 'shkilev.online' will expire on Thu 11 Feb 2021 12:41:56 GMT +0000. [14:22:51] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.78, 20.94, 19.23 [14:24:46] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.90, 19.34, 18.84 [14:33:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.88, 19.99, 19.11 [14:35:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.17, 19.51, 19.05 [14:44:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.98, 22.01, 20.24 [14:46:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.49, 19.18, 19.41 [15:04:04] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.96, 21.36, 19.53 [15:06:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.84, 19.23, 18.94 [15:22:49] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.81, 21.14, 20.07 [15:24:44] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.29, 17.57, 18.90 [15:32:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.19, 19.92, 19.40 [15:34:21] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.07, 19.16, 19.19 [15:43:02] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.88, 23.27, 20.81 [15:44:57] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.07, 20.08, 19.96 [15:47:30] PROBLEM - shkilev.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for shkilev.online could not be found [16:21:52] RECOVERY - shkilev.online - reverse DNS on sslhost is OK: rDNS OK - shkilev.online reverse DNS resolves to cp9.miraheze.org [16:24:01] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.46, 21.28, 18.96 [16:26:02] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.93, 20.39, 18.87 [16:28:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.68, 21.12, 19.26 [16:30:01] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.07, 18.81, 18.64 [16:44:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.22, 21.03, 19.78 [16:46:01] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.19, 19.60, 19.46 [16:52:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.41, 21.29, 20.23 [16:56:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.39, 24.29, 21.56 [17:03:23] PROBLEM - bharatwiki.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bharatwiki.online could not be found [17:03:28] PROBLEM - poserdazfreebies.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for poserdazfreebies.orain.org could not be found [17:03:29] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [17:03:30] PROBLEM - animefeet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for animefeet.miraheze.org could not be found [17:03:30] PROBLEM - wiki.twilightsignal.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.twilightsignal.com could not be found [17:05:05] !log reception@jobrunner1:/srv/mediawiki/w/extensions/ManageWiki/maintenance$ sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki idleonwiki --overwrite [17:05:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --wiki dragalialostwiki /home/reception/dragalialostgamepediacom-20201105-history-2.xml [17:06:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:15] RECOVERY - bharatwiki.online - reverse DNS on sslhost is OK: rDNS OK - bharatwiki.online reverse DNS resolves to cp6.miraheze.org [17:10:25] RECOVERY - wiki.twilightsignal.com - reverse DNS on sslhost is OK: rDNS OK - wiki.twilightsignal.com reverse DNS resolves to cp7.miraheze.org [17:10:26] RECOVERY - animefeet.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - animefeet.miraheze.org reverse DNS resolves to cp7.miraheze.org [17:10:28] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: rDNS OK - www.lab612.at reverse DNS resolves to cp6.miraheze.org [17:10:29] RECOVERY - poserdazfreebies.orain.org - reverse DNS on sslhost is OK: rDNS OK - poserdazfreebies.orain.org reverse DNS resolves to cp6.miraheze.org [17:26:02] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.42, 20.63, 23.45 [17:30:35] [02mw-config] 07Universal-Omega opened pull request 03#3433: Migrate $wgTmhEnableMp4Uploads to ManageWikiSettings - 13https://git.io/JkOUe [17:33:33] [02mw-config] 07Universal-Omega synchronize pull request 03#3433: Migrate $wgTmhEnableMp4Uploads to ManageWikiSettings - 13https://git.io/JkOUe [17:42:02] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.40, 21.17, 22.04 [17:44:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.26, 22.02, 22.25 [17:48:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.33, 24.43, 23.21 [17:50:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.37, 23.23, 22.91 [17:55:04] [02ManageWiki] 07Universal-Omega synchronize pull request 03#200: Added per-namespace override support for $wgManageWikiNamespacesAdditional - 13https://git.io/Jkkdn [17:56:46] [02ManageWiki] 07JohnFLewis reviewed pull request 03#200 commit - 13https://git.io/JkOT1 [18:00:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.32, 16.77, 19.94 [18:02:32] [02ManageWiki] 07Universal-Omega synchronize pull request 03#200: Added per-namespace override support for $wgManageWikiNamespacesAdditional - 13https://git.io/Jkkdn [18:02:56] [02ManageWiki] 07Universal-Omega reviewed pull request 03#200 commit - 13https://git.io/JkOku [18:06:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 34.83, 25.66, 22.51 [18:08:56] [02ManageWiki] 07Universal-Omega synchronize pull request 03#200: Added per-namespace override support for $wgManageWikiNamespacesAdditional - 13https://git.io/Jkkdn [18:09:46] [02mw-config] 07dmehus opened pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOIq [18:09:57] [02ManageWiki] 07JohnFLewis closed pull request 03#200: Added per-namespace override support for $wgManageWikiNamespacesAdditional - 13https://git.io/Jkkdn [18:09:59] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkOIm [18:10:00] [02miraheze/ManageWiki] 07Universal-Omega 03b46d8f5 - Added per-namespace override support for $wgManageWikiNamespacesAdditional (#200) * Added per-namespace override support for $wgManageWikiNamespacesAdditional * tabbing fix * Remove foreach * add $a['overridedefault']['default'] key for default overridedefault * Simplify * Added back is_array check I accidentally removed. [18:10:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.49, 23.44, 22.42 [18:11:05] [02mw-config] 07Universal-Omega commented on pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOIn [18:11:10] [02mw-config] 07Reception123 closed pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOIq [18:11:11] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkOIc [18:11:13] [02miraheze/mw-config] 07dmehus 037b08902 - Add help subdomain and additional numerical values to subdomain subdomain to CreateWikiBlacklist (#3434) Both are common sense additions, much like `noc`, `outreach`, `gazetteer`, and `wikitech` above, in that `help` is a potential, though unneeded now, future Miraheze community wiki. As well, adding numerical modifiers to the `subdomain` subdomain is also helpful. [18:11:57] [02mw-config] 07Universal-Omega edited a comment on pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOIn [18:12:05] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 23.45, 24.03, 22.78 [18:13:12] [02mw-config] 07dmehus commented on pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOIg [18:13:56] [02mw-config] 07Universal-Omega commented on pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOI6 [18:14:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.52, 22.25, 22.23 [18:14:52] [02mw-config] 07Universal-Omega synchronize pull request 03#3413: Add $wgPreloaderSource to ManageWikiNamespaces (T4943) - 13https://git.io/JkLBt [18:15:17] [02mw-config] 07dmehus commented on pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOI1 [18:15:51] [02mw-config] 07dmehus edited pull request 03#3434: Add help subdomain and additional numerical values to subdomain subdo… - 13https://git.io/JkOIq [18:16:34] [02mw-config] 07dmehus edited pull request 03#3434: Add `help` subdomain and additional numerical values to `subdomain` in CreateWikiBlacklist - 13https://git.io/JkOIq [18:16:48] [02mw-config] 07Universal-Omega commented on pull request 03#3434: Add `help` subdomain and additional numerical values to `subdomain` in CreateWikiBlacklist - 13https://git.io/JkOIF [18:17:01] [02mw-config] 07dmehus edited pull request 03#3434: Add `help` subdomain and additional numerical values to `subdomain` subdomain in CreateWikiBlacklist - 13https://git.io/JkOIq [18:17:34] [02mw-config] 07dmehus commented on pull request 03#3434: Add `help` subdomain and additional numerical values to `subdomain` subdomain in CreateWikiBlacklist - 13https://git.io/JkOIx [18:19:33] [02mw-config] 07dmehus edited a comment on pull request 03#3434: Add `help` subdomain and additional numerical values to `subdomain` subdomain in CreateWikiBlacklist - 13https://git.io/JkOIx [18:20:46] [02mw-config] 07dmehus commented on pull request 03#3434: Add `help` subdomain and additional numerical values to `subdomain` subdomain in CreateWikiBlacklist - 13https://git.io/JkOLL [18:26:01] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.70, 24.60, 22.40 [18:30:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [18:30:37] [02mediawiki] 07Universal-Omega opened pull request 03#495: Updating ManageWiki - 13https://git.io/JkOtJ [18:31:20] PROBLEM - animatedfeet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for animatedfeet.miraheze.org could not be found [18:32:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:33:29] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 9.14, 6.89, 4.97 [18:34:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:34:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:35:23] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.11, 5.87, 4.83 [18:35:37] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 6.89, 6.16, 4.94 [18:36:02] Why are we getting that many icinga alerts? (getting them all day yesterday and again quite a few today) [18:36:53] I think we get load warnings most days [18:37:36] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 4.48, 5.47, 4.83 [18:37:38] Yes, but not quite as much. (or maybe I never payed attention) [18:38:09] RECOVERY - animatedfeet.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - animatedfeet.miraheze.org reverse DNS resolves to cp7.miraheze.org [18:38:42] At this point icinga is white noise to me, so I don't know if it's gotten worse [18:47:36] [02mw-config] 07paladox closed pull request 03#3413: Add $wgPreloaderSource to ManageWikiNamespaces (T4943) - 13https://git.io/JkLBt [18:47:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkOqr [18:47:39] [02miraheze/mw-config] 07Universal-Omega 03d4b2ddb - Add $wgPreloaderSource to ManageWikiNamespaces (T4943) (#3413) * Add $wgPreloaderSource to ManageWikiNamespaces (T4943) * fix * add 'from' * add default key [18:48:27] [02mediawiki] 07paladox closed pull request 03#495: Updating ManageWiki - 13https://git.io/JkOtJ [18:48:28] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JkOqM [18:48:30] [02miraheze/mediawiki] 07Universal-Omega 037a1bf87 - Updating ManageWiki [18:48:31] [02miraheze/mediawiki] 07paladox 0376f3e25 - Merge pull request #495 from Universal-Omega/REL1_35 Updating ManageWiki [18:52:22] @Universal_Omega yeah what @sario528 said, though I do agree they seem to be more frequent the past couple days. They do self-correct usually, so it's probably not anything to worry about, but I do wonder if it's relayed to the very long page load times I'm encountering on Meta (2000ms or greater, on average)? [18:52:52] s/relayed/related [18:52:52] dmehus meant to say: @Universal_Omega yeah what @sario528 said, though I do agree they seem to be more frequent the past couple days. They do self-correct usually, so it's probably not anything to worry about, but I do wonder if it's related to the very long page load times I'm encountering on Meta (2000ms or greater, on average)? [19:00:43] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:00:50] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:01:00] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:01:11] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:01:15] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:01:37] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:01:44] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 7.36, 4.57, 3.54 [19:01:49] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:02:02] [02mediawiki] 07Universal-Omega opened pull request 03#496: Removed WhoIsWatching - 13https://git.io/JkOYm [19:02:43] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:02:49] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:02:57] [02mediawiki] 07paladox closed pull request 03#496: Removed WhoIsWatching - 13https://git.io/JkOYm [19:02:58] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:02:58] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-2/±2] 13https://git.io/JkOYn [19:03:00] [02miraheze/mediawiki] 07Universal-Omega 03d330ce6 - Removed WhoIsWatching [19:03:01] [02miraheze/mediawiki] 07paladox 03c20b3a6 - Merge pull request #496 from Universal-Omega/REL1_35 Removed WhoIsWatching [19:03:07] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:03:35] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:03:44] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.54, 3.71, 3.35 [19:03:48] I reported these in #general on Discord today- > "I just got this error while trying to publish a translation for [[mh:template:Template Wiki:General Disclaimer]]- > Publishing the translation failed: Unknown error occurred." https://cdn.discordapp.com/attachments/407537962553966603/776810916838375454/Screenshot_20201113-200829_Chrome.jpg > I am sure I have a good internet connection. > This happened while trying to reply to a topic. > [19:03:48] Everything else is loading fast and fine. > . Could these be related? [19:05:12] (Both were resolved after retrying and reloading the page.) [19:07:14] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [19:07:48] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:20:04] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.15, 20.92, 23.38 [19:22:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.63, 22.76, 23.77 [19:22:36] [02mw-config] 07Universal-Omega opened pull request 03#3435: Migrate $wgFriendingEnabled to ManageWikiSettings - 13https://git.io/JkOO9 [19:23:17] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.73, 6.56, 5.75 [19:24:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.59, 23.28, 23.86 [19:25:16] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 4.97, 6.07, 5.68 [19:25:32] [02mw-config] 07Universal-Omega synchronize pull request 03#3435: Migrate $wgFriendingEnabled to ManageWikiSettings - 13https://git.io/JkOO9 [19:30:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.10, 24.99, 24.11 [19:32:45] [02mw-config] 07Universal-Omega opened pull request 03#3436: Migrate Timeless configs to ManageWikiSettings - 13https://git.io/JkO3E [19:35:11] [02mw-config] 07Universal-Omega synchronize pull request 03#3436: Migrate Timeless configs to ManageWikiSettings - 13https://git.io/JkO3E [19:42:21] PROBLEM - mon1 Current Load on mon1 is WARNING: WARNING - load average: 3.60, 2.79, 2.46 [19:42:32] [02mw-config] 07Universal-Omega synchronize pull request 03#3436: Migrate Timeless configs to ManageWikiSettings - 13https://git.io/JkO3E [19:44:19] RECOVERY - mon1 Current Load on mon1 is OK: OK - load average: 2.69, 2.70, 2.46 [19:53:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:105a::10/cpweb [19:54:15] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 9.66, 8.64, 5.89 [19:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkOGv [19:55:13] [02miraheze/services] 07MirahezeSSLBot 03403ed32 - BOT: Updating services config for wikis [19:58:24] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.44, 7.23, 5.33 [19:59:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:00:18] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.60, 6.63, 5.36 [20:06:03] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 3.78, 7.25, 7.64 [20:11:54] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.88, 5.12, 6.62 [20:20:58] [02mw-config] 07Universal-Omega opened pull request 03#3437: Cleanup LocalSettings.php - 13https://git.io/JkOnq [20:40:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.65, 20.35, 23.53 [20:52:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.91, 24.11, 23.76 [20:54:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.87, 23.41, 23.59 [20:56:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.21, 24.20, 23.88 [21:00:03] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.97, 23.89, 23.87 [21:02:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.55, 26.29, 24.72 [21:06:02] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.33, 23.47, 23.95 [21:20:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.55, 17.70, 20.30 [21:20:06] @HeartsDo Do you know how to set up Twinkle? [21:20:27] A few users said you could help me [21:24:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.27, 23.15, 22.03 [21:26:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.71, 22.76, 22.07 [21:28:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.88, 24.63, 22.75 [21:30:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.76, 22.52, 22.19 [21:38:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 35.01, 26.41, 23.70 [21:41:30] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [21:41:30] PROBLEM - wiki.worlduniversityandschool.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worlduniversityandschool.org could not be found [21:41:31] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [21:41:32] PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ga could not be found [21:41:32] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [21:46:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.31, 21.17, 23.00 [21:48:11] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: rDNS OK - warmage.necrocity.co.uk reverse DNS resolves to cp6.miraheze.org [21:48:14] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: rDNS OK - wiki.nowchess.org reverse DNS resolves to cp7.miraheze.org [21:48:16] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp7.miraheze.org [21:48:17] RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: rDNS OK - miraheze.ga reverse DNS resolves to cp6.miraheze.org [21:48:30] RECOVERY - wiki.worlduniversityandschool.org - reverse DNS on sslhost is OK: rDNS OK - wiki.worlduniversityandschool.org reverse DNS resolves to cp6.miraheze.org [21:54:01] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.47, 16.43, 19.89 [22:08:04] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.32, 22.15, 20.61 [22:10:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkO0b [22:10:22] [02miraheze/services] 07MirahezeSSLBot 03d5ada52 - BOT: Updating services config for wikis [22:14:06] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.11, 22.16, 21.42 [22:16:07] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 91s [22:18:06] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [22:20:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.89, 17.50, 19.60 [22:20:24] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JkOEa [22:20:26] [02miraheze/services] 07MirahezeSSLBot 0319ad46c - BOT: Updating services config for wikis [23:32:50] https://phabricator.miraheze.org/T6431 - cp9 extremely latent for a number of users on Discord, authored by SCVSlalom, assigned to None, Priority: High, Status: Open [23:32:52] https://phabricator.miraheze.org/T6411 - Connection to restbase (and possibly parsoid) failing on services*, authored by RhinosF1, assigned to None, Priority: High, Status: Open [23:32:53] https://phabricator.miraheze.org/T6391 - Fatal exception of type "Wikimedia \ Rdbms \ DBQueryError" occurring on shiropediawiki, authored by J-Josyu, assigned to Reception123, Priority: High, Status: Open [23:37:55] !log upgrade puppet-agent puppetdb puppetdb-termini puppetserver on puppet2 [23:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:19] !log upgrade puppet-agent on cp9 [23:39:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:07] !log upgrade puppet-agent on mw*, jobrunner* and test2 [23:44:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:45:08] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [23:45:18] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [23:45:41] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown) [23:45:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown) [23:45:53] !log upgrade puppet-agent on cp[367] [23:45:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:11] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 2 minutes ago with 24 failures. Failed resources (up to 3 shown) [23:46:25] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [23:46:35] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 2 minutes ago with 432 failures. Failed resources (up to 3 shown) [23:46:47] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 441 failures. Last run 3 minutes ago with 441 failures. Failed resources (up to 3 shown) [23:46:49] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [23:46:50] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 453 failures. Last run 3 minutes ago with 453 failures. Failed resources (up to 3 shown) [23:46:52] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 447 failures. Last run 2 minutes ago with 447 failures. Failed resources (up to 3 shown) [23:46:59] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 446 failures. Last run 3 minutes ago with 446 failures. Failed resources (up to 3 shown) [23:47:08] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 448 failures. Last run 3 minutes ago with 448 failures. Failed resources (up to 3 shown) [23:47:10] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [23:47:11] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 430 failures. Last run 3 minutes ago with 430 failures. Failed resources (up to 3 shown) [23:47:29] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 455 failures. Last run 3 minutes ago with 455 failures. Failed resources (up to 3 shown) [23:47:42] !log upgrade grafana on mon1 [23:47:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:47:49] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 433 failures. Last run 3 minutes ago with 433 failures. Failed resources (up to 3 shown): File[spiral.wiki_private],File[permanentfuturelab.wiki],File[permanentfuturelab.wiki_private],File[private.revi.wiki] [23:48:03] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 132 failures. Last run 2 minutes ago with 132 failures. Failed resources (up to 3 shown): File[spiral.wiki_private],File[permanentfuturelab.wiki],File[permanentfuturelab.wiki_private],File[private.revi.wiki] [23:48:04] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.15, 21.94, 18.42 [23:48:59] !log upgrade mariadb-client-10.4 mariadb-client-core-10.4 mariadb-common mysql-common libmariadb3 puppet-agent on mon1 [23:49:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:01] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.47, 18.23, 17.47 [23:50:11] !log upgrade puppet-agent on phab1 and rdb* [23:50:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:11] !log upgrade puppet-agent on db* [23:51:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:08] !log upgrade puppet-agent on cloud* and services* [23:52:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:59] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:53:07] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:53:16] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:53:23] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:53:25] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:53:32] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:53:41] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:46] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:48] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:54:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.31, 21.66, 19.32 [23:54:10] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:25] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:32] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:43] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:47] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:49] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:52] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:59] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:07] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:18] !log upgrade puppet-agent on ns*, mail1, gluster* and graylog*. [23:55:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.09, 19.18, 18.68 [23:56:11] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:56:19] !log upgrade puppet-agent on ldap1 [23:56:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:13] [02mw-config] 07Universal-Omega opened pull request 03#3438: Migrate $wgMinervaAlwaysShowLanguageButton to MWS - 13https://git.io/JkOwz