[00:03:55] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 21.61, 25.73, 22.90 [00:09:48] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.00, 20.32, 21.62 [00:15:42] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 11.31, 16.43, 19.69 [00:18:52] miraheze/mw-config/master/8de984d - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/747522671 [00:25:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.45, 20.62, 19.76 [00:27:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.73, 16.74, 18.43 [00:33:15] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.31, 22.83, 20.28 [00:37:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.38, 22.68, 20.75 [00:41:03] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.94, 24.19, 21.84 [00:46:55] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.52, 22.91, 22.48 [00:52:49] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.35, 23.14, 22.28 [00:58:45] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.11, 21.81, 21.95 [01:00:47] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.56, 26.94, 23.84 [01:06:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.30, 22.57, 23.58 [01:12:45] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.33, 22.06, 22.61 [01:34:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.77.107.210/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:34:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:36:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:36:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:52:45] PROBLEM - rdb2 APT on rdb2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:54:52] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [01:55:39] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:55:44] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:32] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:49] PROBLEM - cp9 Stunnel Http for mw5 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:56:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [01:57:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 2607:5300:205:200::2ac4/cpweb [01:57:52] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 5.069 second response time [01:57:53] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw5 [01:57:59] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [01:58:27] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 19603 bytes in 1.065 second response time [01:58:48] RECOVERY - cp9 Stunnel Http for mw5 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15955 bytes in 0.324 second response time [01:58:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:59:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:59:40] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15963 bytes in 0.003 second response time [01:59:48] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [01:59:59] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [02:01:02] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [02:08:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 9.89, 15.75, 22.53 [02:16:45] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 9.57, 14.04, 19.42 [02:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIYuN [02:35:12] [02miraheze/services] 07MirahezeSSLBot 031198289 - BOT: Updating services config for wikis [03:00:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIYaT [03:00:23] [02miraheze/services] 07MirahezeSSLBot 03cc3ca74 - BOT: Updating services config for wikis [06:07:38] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIYA4 [06:07:40] [02miraheze/mediawiki] 07Universal-Omega 0381d84b6 - Updating Cosmos [06:07:41] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [06:09:05] [02mediawiki] 07Universal-Omega opened pull request 03#625: Updating Cosmos - 13https://git.io/JIYAg [06:27:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIYpj [06:27:25] [02miraheze/mediawiki] 07Universal-Omega 03f63fd13 - Updating Cosmos (fix) [06:27:26] [02mediawiki] 07Universal-Omega synchronize pull request 03#625: Updating Cosmos - 13https://git.io/JIYAg [06:30:03] [02mediawiki] 07Reception123 closed pull request 03#625: Updating Cosmos - 13https://git.io/JIYAg [06:30:05] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JIYhl [06:30:06] [02miraheze/mediawiki] 07Universal-Omega 037636c50 - Updating Cosmos (#625) * Updating Cosmos * Updating Cosmos (fix) [06:40:06] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:40:13] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:40:52] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [06:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:58] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:40:59] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:41:09] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:41:14] Where should I ask for a bigdelete? [06:41:35] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:41:40] MadriCR: Stewards' noticeboard is probably best :) [06:41:46] Ty :) [06:41:56] Np [06:42:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIOev [06:42:53] [02miraheze/mw-config] 07Universal-Omega 03cbe0d32 - Add Donate link to the footer, after termsofservice link [06:42:55] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [06:43:02] [02mw-config] 07Universal-Omega opened pull request 03#3521: Add Donate link to the footer, after termsofservice link (T6545) - 13https://git.io/JIOef [06:43:08] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:43:34] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:44:03] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:44:12] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:44:16] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 6.85, 4.20, 2.29 [06:44:58] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:44:59] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:46:03] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIOe8 [06:46:04] [02miraheze/MirahezeMagic] 07Universal-Omega 039082104 - update donate messages [06:46:06] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [06:46:07] [02MirahezeMagic] 07Universal-Omega opened pull request 03#163: update donate messages - 13https://git.io/JIOeB [06:47:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JIOer [06:47:20] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.35, 21.99, 16.96 [06:47:21] [02miraheze/mw-config] 07Universal-Omega 0333aecd6 - update messages [06:47:22] [02mw-config] 07Universal-Omega synchronize pull request 03#3521: Add Donate link to the footer, after termsofservice link (T6545) - 13https://git.io/JIOef [06:49:17] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.57, 19.64, 16.71 [06:53:52] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 33.71, 29.73, 20.57 [06:54:15] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.29, 5.91, 4.45 [06:54:46] [02MirahezeMagic] 07dmehus commented on pull request 03#163: update donate messages - 13https://git.io/JIOvo [06:58:15] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 3.19, 4.96, 4.42 [06:59:51] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JIOfB [06:59:51] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.21, 20.18, 19.43 [06:59:52] [02miraheze/MirahezeMagic] 07Universal-Omega 03f79de58 - remove ! [06:59:54] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#163: update donate messages - 13https://git.io/JIOeB [07:00:37] !log $cw = new CreateWikiJson( 'planetnomadswiki' ); $cw->resetWiki(); [07:00:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:15] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 5.27, 5.55, 4.86 [07:08:16] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 6.94, 5.78, 5.01 [07:16:16] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 5.55, 5.98, 5.47 [07:18:24] !log sudo -u www-data rm /srv/mediawiki/w/cache/planetnomadswiki.json on mw* [07:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:22:15] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 7.49, 6.34, 5.76 [07:26:15] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 4.45, 5.89, 5.76 [07:26:38] Reception123: To enable Extension:ParserFunctions I must open a ticket on Phabricator, right? [07:27:15] MadriCR: that extension is actually enabled globally for all wikis [07:27:20] oh [07:27:32] but you can also enable other extensions yourself via Special:ManageWiki/extensions [07:27:35] I'm dumb [07:27:39] thank you [07:27:49] no problem :) [07:28:11] Also Scribunto? [07:28:21] (I'm reading https://www.mediawiki.org/wiki/Help:Templates/es#Copying_from_one_wiki_to_another) [07:28:22] [ Help:Plantillas - MediaWiki ] - www.mediawiki.org [07:29:11] MadriCR: Scribunto is also globally enabled. You can see all the currently enabled extensions on Special:Version [07:29:19] great, thank you [07:33:00] @Reception123: Don't forget jobrunners and not sure how much we care for test2 [07:35:06] @RhinosF1 yeah I figured out what I did wrong after [07:35:22] (only ran the eval.php on jobrunner1) [07:38:15] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 3.11, 4.20, 4.89 [08:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIOGy [08:25:12] [02miraheze/services] 07MirahezeSSLBot 038105ae1 - BOT: Updating services config for wikis [09:44:57] Can someone create the wiki I propose it? [09:45:56] MH-Discord [09:47:08] Are they any sysop here? [09:47:14] Plz [09:48:45] Hello? [10:36:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JIOVN [10:36:33] [02miraheze/puppet] 07paladox 035f33bb1 - puppet: Tweak settings [10:36:34] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [10:36:36] [02puppet] 07paladox opened pull request 03#1539: puppet: Tweak settings - 13https://git.io/JIOVA [10:42:53] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:42:53] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:42:54] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:42:56] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:42:57] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:10] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:15] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:17] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:18] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:21] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:30] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:30] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 minutes ago with 20 failures. Failed resources (up to 3 shown) [10:43:34] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:34] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:37] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:37] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:39] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:47] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:52] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:43:54] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:03] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:10] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:13] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:23] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:25] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:26] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:35] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:37] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 3 minutes ago with 18 failures. Failed resources (up to 3 shown) [10:44:38] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:40] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:44:40] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:45:29] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [10:45:34] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [10:45:38] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [10:45:51] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [10:45:53] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:46:08] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [10:46:23] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [10:46:25] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:46:37] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:46:38] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [10:46:40] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [10:46:40] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:46:52] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:46:53] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:46:56] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [10:47:16] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:47:17] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [10:47:18] PROBLEM - puppet2 Puppet on puppet2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 57 seconds ago with 0 failures [10:47:20] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:47:38] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:47:46] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:51:44] Can someone help me? [10:51:44] shadi: Pinging Reception123, Zppix, PuppyKun, Voidwalker, or RhinosF1 who might be able to help you. [10:51:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:51:52] !log depool mw[45], change cpu cores to 4 and repool [10:51:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:26] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:52:37] Can someone help me? [10:52:37] shadi: Pinging Reception123, Zppix, PuppyKun, Voidwalker, or RhinosF1 who might be able to help you. [10:52:53] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [10:53:09] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:53:30] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:53:34] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:53:39] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:54:04] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:54:07] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [10:54:09] Can someone help me? [10:54:10] shadi: Pinging Reception123, Zppix, PuppyKun, Voidwalker, or RhinosF1 who might be able to help you. [10:54:13] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:54:32] PROBLEM - mw4 Check Gluster Clients on mw4 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [10:54:34] shadi hi, what do you need help with? [10:55:01] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 37 seconds ago with 0 failures [10:55:29] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [10:55:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [10:55:38] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [10:55:42] I developp a wiki in xampp but xampp crash and I want to tansfer the wiki on it? [10:55:53] Miraheze [10:56:07] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [10:56:30] Xa [10:56:32] RECOVERY - mw4 Check Gluster Clients on mw4 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [10:56:54] I can acess to the wiki on xampp anymore [10:58:57] PROBLEM - mw5 Check Gluster Clients on mw5 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [10:59:00] i am not sure what you mean by transfer shadi [10:59:32] I want to take away the pages on a wiki in miraheze [11:00:57] RECOVERY - mw5 Check Gluster Clients on mw5 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [11:01:29] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [11:01:30] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [11:01:39] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [11:02:03] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [11:02:46] shadi oh, you can do an export of your wiki then ask us to import using https://phabricator.miraheze.org [11:02:47] [ Main Page ] - phabricator.miraheze.org [11:06:33] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [11:06:49] !log decrease graylogs cores to 2 [11:06:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:09:23] !log increase puppet2 cores to 4 and ram to 6gb [11:09:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:09:26] PROBLEM - graylog1 HTTPS on graylog1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 311 bytes in 0.006 second response time [11:11:30] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: connect to address 51.89.160.129 port 5666: Connection refusedconnect to host 51.89.160.129 port 5666: Connection refused [11:13:29] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [11:13:40] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:13:41] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:13:43] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:13:51] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:13:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:13:59] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:08] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:14:16] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:18] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:28] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:32] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:34] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:41] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:44] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:48] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:52] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:14:56] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:14:57] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:01] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:01] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:12] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:16] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:19] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:20] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:30] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:31] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:33] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:34] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:37] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:37] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [11:16:49] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [11:16:55] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [11:17:14] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:17:19] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [11:17:20] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [11:17:29] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:17:33] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:17:35] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [11:17:37] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [11:17:37] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:17:45] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [11:17:48] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:17:53] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:12] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:13] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [11:18:26] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:26] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [11:18:33] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:38] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:52] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:01] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:01] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:08] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:28] RECOVERY - graylog1 HTTPS on graylog1 is OK: HTTP OK: HTTP/1.1 200 OK - 1670 bytes in 2.337 second response time [11:19:29] RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:34] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:19:39] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:20:17] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 7.29, 3.89, 1.64 [11:20:50] !log depool mw[67], change cpu cores to 4 and repool [11:20:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:22:15] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.59, 3.60, 1.80 [11:23:57] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw6 [11:24:06] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 25.77, 20.45, 16.08 [11:24:30] RECOVERY - graylog1 Current Load on graylog1 is OK: OK - load average: 1.70, 2.99, 1.83 [11:24:57] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [11:25:43] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:25:53] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [11:25:54] PROBLEM - hw.gyaanipedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hw.gyaanipedia.co.in could not be found [11:25:54] PROBLEM - wiki.thehall.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thehall.xyz could not be found [11:25:55] PROBLEM - wiki.xfbs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xfbs.net could not be found [11:25:56] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [11:25:56] PROBLEM - evenair.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for evenair.tk could not be found [11:25:57] PROBLEM - www.reviwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.reviwiki.info could not be found [11:26:02] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.37, 18.78, 15.98 [11:26:49] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:27:39] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:28:48] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [11:30:38] [02MirahezeMagic] 07Reception123 closed pull request 03#163: update donate messages - 13https://git.io/JIOeB [11:30:40] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIO1r [11:30:41] [02miraheze/MirahezeMagic] 07Universal-Omega 031c282d0 - update donate messages (#163) * update donate messages * remove ! [11:30:56] [02MirahezeMagic] 07Reception123 deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [11:30:58] [02miraheze/MirahezeMagic] 07Reception123 deleted branch 03Universal-Omega-patch-1 [11:32:35] RECOVERY - wiki.xfbs.net - reverse DNS on sslhost is OK: rDNS OK - wiki.xfbs.net reverse DNS resolves to cp6.miraheze.org [11:32:38] RECOVERY - hw.gyaanipedia.co.in - reverse DNS on sslhost is OK: rDNS OK - hw.gyaanipedia.co.in reverse DNS resolves to cp7.miraheze.org [11:32:38] RECOVERY - wiki.thehall.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.thehall.xyz reverse DNS resolves to cp7.miraheze.org [11:32:38] RECOVERY - evenair.tk - reverse DNS on sslhost is OK: rDNS OK - evenair.tk reverse DNS resolves to cp6.miraheze.org [11:32:40] RECOVERY - miraheze.tk - reverse DNS on sslhost is OK: rDNS OK - miraheze.tk reverse DNS resolves to cp7.miraheze.org [11:32:46] RECOVERY - www.reviwiki.info - reverse DNS on sslhost is OK: rDNS OK - www.reviwiki.info reverse DNS resolves to cp6.miraheze.org [12:00:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JIO99 [12:00:09] [02miraheze/mw-config] 07paladox 03d1a47ee - Set GlobalTitleFail to error [12:17:53] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JIO7x [12:17:54] [02miraheze/mediawiki] 07Reception123 03f496da0 - update MirahezeMagic [12:30:02] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [12:30:07] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:30:17] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:30:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:30:57] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:31:10] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:31:13] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:31:35] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:33:08] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [12:33:10] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [12:33:29] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 10.06, 7.23, 5.20 [12:33:34] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [12:33:57] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.92, 7.91, 5.38 [12:34:06] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [12:34:12] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:34:17] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.27, 19.51, 15.43 [12:34:56] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:35:27] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.29, 6.82, 5.30 [12:35:55] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 5.98, 7.02, 5.36 [12:36:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.52, 19.11, 15.80 [12:36:15] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 6.11, 4.77, 2.73 [12:37:23] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.42, 6.37, 5.32 [12:37:56] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.79, 6.26, 5.28 [12:43:16] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 14.28, 10.27, 7.16 [12:43:58] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 12.74, 9.93, 7.09 [12:44:08] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.85, 22.24, 18.16 [12:44:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb [12:44:51] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [12:46:04] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.74, 21.29, 18.25 [12:46:49] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 19599 bytes in 0.926 second response time [12:48:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:48:53] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:49:07] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 5.85, 7.85, 7.10 [12:49:57] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.36, 19.58, 18.32 [12:53:03] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.02, 8.17, 7.40 [12:53:53] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.57, 21.30, 19.45 [12:55:03] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 4.80, 7.42, 7.26 [12:55:21] !log set puppet2 cpu cores to 3 (down from 4) [12:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:53] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.42, 17.98, 18.46 [12:55:58] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 4.70, 6.77, 7.14 [12:57:55] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 2.21, 5.15, 6.50 [12:58:17] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.46, 3.94, 3.91 [12:59:02] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 3.03, 5.01, 6.30 [13:04:15] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.68, 4.43, 4.10 [13:12:40] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.82, 5.92, 5.81 [13:14:35] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.83, 6.07, 5.89 [13:16:08] PROBLEM - wiki.fourta.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fourta.org could not be found [13:16:09] PROBLEM - wooriwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wooriwiki.com could not be found [13:16:10] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [13:16:12] PROBLEM - innersphere.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for innersphere.tk could not be found [13:16:12] PROBLEM - wiki.sketch.town - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sketch.town could not be found [13:16:15] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.89, 3.84, 3.98 [13:17:10] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [13:17:22] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [13:17:38] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [13:17:41] PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 178 failures. Last run 2 minutes ago with 178 failures. Failed resources (up to 3 shown) [13:18:04] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [13:18:14] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown) [13:18:57] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown) [13:20:17] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CRITICAL - load average: 4.52, 4.06, 4.01 [13:20:39] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 142 failures. Last run 3 minutes ago with 142 failures. Failed resources (up to 3 shown): File[wiki.zymonic.com_private],File[wiki.mxlinuxusers.de],File[wiki.mxlinuxusers.de_private],File[cyberlaw.ccdcoe.org] [13:22:50] RECOVERY - wiki.fourta.org - reverse DNS on sslhost is OK: rDNS OK - wiki.fourta.org reverse DNS resolves to cp7.miraheze.org [13:22:52] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp6.miraheze.org [13:22:52] RECOVERY - wooriwiki.com - reverse DNS on sslhost is OK: rDNS OK - wooriwiki.com reverse DNS resolves to cp7.miraheze.org [13:23:00] RECOVERY - innersphere.tk - reverse DNS on sslhost is OK: rDNS OK - innersphere.tk reverse DNS resolves to cp7.miraheze.org [13:23:00] RECOVERY - wiki.sketch.town - reverse DNS on sslhost is OK: rDNS OK - wiki.sketch.town reverse DNS resolves to cp6.miraheze.org [13:26:16] PROBLEM - graylog1 Current Load on graylog1 is WARNING: WARNING - load average: 3.78, 3.91, 3.98 [13:26:23] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [13:26:25] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:26:34] PROBLEM - puppet2 puppetserver on puppet2 is CRITICAL: connect to address 51.89.160.129 and port 8140: Connection refused [13:26:37] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown) [13:26:38]