[00:00:02] !log running lc on mw* [00:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [04:02:46] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.00, 3.08, 1.31 [04:06:45] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.09, 1.43, 1.03 [05:02:39] PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 3.80, 2.32, 1.12 [05:02:45] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.27, 2.06, 0.95 [05:04:38] RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 0.79, 1.66, 1.02 [05:04:45] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.50, 1.41, 0.85 [06:45:55] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.2-apcu],Package[php7.2-redis] [06:53:55] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:02:20] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.60, 3.35, 2.88 [07:04:19] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.05, 3.26, 2.91 [08:34:59] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [08:35:45] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 189 failures. Last run 2 minutes ago with 189 failures. Failed resources (up to 3 shown) [08:35:58] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [08:36:41] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [08:37:14] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 210 failures. Last run 3 minutes ago with 210 failures. Failed resources (up to 3 shown): File[wiki.ngscott.net_private],File[wiki.ombre.io],File[wiki.ombre.io_private],File[wiki.dobots.nl] [08:43:47] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:43:57] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:44:44] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:44:54] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [08:45:15] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:45:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:02:46] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.26, 2.55, 1.10 [09:03:47] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:45] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.73, 1.81, 1.01 [09:06:45] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.22, 1.29, 0.91 [10:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCEw [10:25:14] [02miraheze/services] 07MirahezeSSLBot 034012adf - BOT: Updating services config for wikis [12:02:53] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.09, 3.09, 1.41 [12:03:21] I feel you cp4 [12:06:48] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.27, 1.50, 1.13 [14:49:48] * Hispano76 saluda [16:32:26] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40276 MB (10% inode=97%); [17:18:05] PROBLEM - secularknowledge.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:13] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:13] PROBLEM - ml.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:17] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:43] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CRITICAL - load average: 17.31, 9.50, 3.75 [17:20:07] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Tue 04 Jun 2019 08:02:30 PM GMT +0000. [17:20:08] RECOVERY - secularknowledge.com - LetsEncrypt on sslhost is OK: OK - Certificate 'secularknowledge.com' will expire on Wed 05 Jun 2019 04:02:01 PM GMT +0000. [17:20:09] RECOVERY - ml.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'ml.gyaanipedia.co.in' will expire on Wed 05 Jun 2019 03:55:53 PM GMT +0000. [17:20:14] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18948 bytes in 0.032 second response time [17:32:35] PROBLEM - misc1 Current Load on misc1 is WARNING: WARNING - load average: 1.52, 1.37, 1.96 [17:36:35] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.17, 0.74, 1.56 [18:44:50] Hello all [18:46:50] hi [18:48:22] I am currently working on a Global CheckUser extension for miraheze. I am open to any suggestions [19:08:02] PROBLEM - www.programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:18] PROBLEM - wikipuk.cl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:19] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:19] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:19] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:19] PROBLEM - documentation.aqfer.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:19] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:19] PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:20] PROBLEM - b1tcash.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:20] PROBLEM - sa.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:21] PROBLEM - studentwiki.ddns.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:21] PROBLEM - la.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:22] PROBLEM - wiki.ameciclo.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:23] PROBLEM - misc2 Redis Process on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:23] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:34] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:59] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Tue 04 Jun 2019 08:23:06 PM GMT +0000. [19:10:02] RECOVERY - wikipuk.cl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipuk.cl' will expire on Tue 04 Jun 2019 04:30:05 PM GMT +0000. [19:10:02] RECOVERY - wiki.kourouklides.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kourouklides.com' will expire on Tue 04 Jun 2019 08:02:30 PM GMT +0000. [19:10:06] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 05 Jun 2019 03:27:39 PM GMT +0000. [19:10:08] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [19:10:08] RECOVERY - documentation.aqfer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'documentation.aqfer.com' will expire on Wed 05 Jun 2019 03:36:28 PM GMT +0000. [19:10:12] RECOVERY - b1tcash.org - LetsEncrypt on sslhost is OK: OK - Certificate 'b1tcash.org' will expire on Wed 05 Jun 2019 02:40:42 PM GMT +0000. [19:10:13] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1331 bytes in 0.650 second response time [19:10:13] RECOVERY - wiki.ngscott.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ngscott.net' will expire on Wed 05 Jun 2019 01:16:18 PM GMT +0000. [19:10:14] RECOVERY - sa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'sa.gyaanipedia.co.in' will expire on Wed 05 Jun 2019 03:02:42 PM GMT +0000. [19:10:15] RECOVERY - la.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'la.gyaanipedia.co.in' will expire on Thu 04 Jul 2019 12:07:35 AM GMT +0000. [19:10:15] RECOVERY - studentwiki.ddns.net - LetsEncrypt on sslhost is OK: OK - Certificate 'studentwiki.ddns.net' will expire on Wed 05 Jun 2019 01:12:11 PM GMT +0000. [19:10:18] RECOVERY - www.mywiki.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'mywiki.arrexventures.com' will expire on Wed 24 Jul 2019 11:59:59 PM GMT +0000. [19:10:18] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Thu 04 Jul 2019 12:04:49 AM GMT +0000. [19:10:19] RECOVERY - wiki.ameciclo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ameciclo.org' will expire on Tue 04 Jun 2019 08:09:27 PM GMT +0000. [19:10:19] RECOVERY - wiki.bonessdiving.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bonessdiving.club' will expire on Wed 05 Jun 2019 01:49:31 PM GMT +0000. [19:10:20] RECOVERY - misc2 Redis Process on misc2 is OK: PROCS OK: 1 process with args 'redis-server' [19:10:23] RECOVERY - wiki.hackdown.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hackdown.org' will expire on Tue 04 Jun 2019 04:10:11 PM GMT +0000. [19:10:24] RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Thu 01 Aug 2019 04:24:39 PM GMT +0000. [19:10:26] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Wed 26 Jun 2019 09:59:51 PM GMT +0000. [19:10:27] RECOVERY - disabled.life - LetsEncrypt on sslhost is OK: OK - Certificate 'disabled.life' will expire on Tue 04 Jun 2019 07:33:19 PM GMT +0000. [19:10:29] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 05 Jun 2019 04:09:38 PM GMT +0000. [19:10:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:10:35] PROBLEM - misc1 Current Load on misc1 is CRITICAL: CRITICAL - load average: 3.27, 1.83, 0.85 [19:12:35] RECOVERY - misc1 Current Load on misc1 is OK: OK - load average: 0.55, 1.27, 0.76 [20:14:56] Hello Folks, I am trying to unclose a wiki that I am slow with. I get an error after filling out the reason why I want it unclosed and get this error message, any ideas! [c5ccbe080abc0692fb83988f] 2019-05-10 20:12:26: Fatal exception of type "Error" [20:18:02] MarkDilley: Hi, I'll take a look [20:18:13] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCXC [20:18:14] [02miraheze/ManageWiki] 07JohnFLewis 0376900f2 - closed->closure [20:18:56] JohnLewis already knows what it is :P [20:19:00] was just about to say [20:19:05] /wiki/Special:ManageWiki Error from line 794 of /srv/mediawiki/w/extensions/ManageWiki/includes/formFactory/ManageWikiFormFactoryBuilder.php: Call to undefined method RemoteWiki::getClosedDate() [20:19:46] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjCXW [20:19:47] [02miraheze/mediawiki] 07JohnFLewis 03df53bae - update MW [20:21:20] Thanks all for checking into [20:25:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCX4 [20:25:14] [02miraheze/services] 07MirahezeSSLBot 03686a7d1 - BOT: Updating services config for wikis [20:30:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCXu [20:30:16] [02miraheze/services] 07MirahezeSSLBot 03f2f5fd9 - BOT: Updating services config for wikis [20:35:14] Thank you, issue resolved. [21:29:42] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjC1z [21:29:43] [02miraheze/puppet] 07Southparkfan 035b01f7a - Assign vpncloud ip for ns1 [21:30:08] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjC1g [21:30:10] [02miraheze/puppet] 07Southparkfan 03bfb3c18 - Assign vpncloud ip for puppet1 [21:30:38] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjC12 [21:30:40] [02miraheze/puppet] 07Southparkfan 033da8935 - Add vpncloud role on ns1/puppet1 [21:32:48] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjC1K [21:32:49] [02miraheze/puppet] 07Southparkfan 0337bd955 - Firewall rules for vpncloud puppet1/ns1 [21:35:43] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjC1y [21:35:44] [02miraheze/puppet] 07Southparkfan 03b9fc549 - Add ns1/puppet1 to vpncloud configuration, for real [21:40:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:41:01] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:41:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [21:41:42] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [21:43:42] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:43:47] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [21:45:58] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [21:46:24] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [21:48:14] !log removed 'citoid' extension from mar9122wiki - T4356 [21:48:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:50:38] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:50:38] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:38] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:38] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:39] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:39] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:40] PROBLEM - mw3 SSH on mw3 is CRITICAL: connect to address 81.4.121.113 and port 22: Connection refused [21:50:40] PROBLEM - mw3 Puppet on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:48] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:48] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:48] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:48] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:50:48] PROBLEM - mw3 SSH on mw3 is CRITICAL: connect to address 81.4.121.113 and port 22: Connection refused [21:50:49] PROBLEM - mw3 Puppet on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:49] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:50] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:50:50] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:51:47] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:53:57] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:54:25] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:17:54] !log hacking User.php on mw1 [22:17:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:18:54] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.81, 6.97, 4.46 [22:22:53] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.10, 6.87, 5.07 [22:24:15] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 51954 MB (67% inode=99%); [22:24:34] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 51 minutes ago with 0 failures [22:24:46] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.015 second response time [22:24:53] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.81, 6.28, 5.08 [22:25:08] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [22:25:43] RECOVERY - mw3 SSH on mw3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [22:26:03] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 1.80, 0.63, 0.22 [22:26:10] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 5 processes with command name 'php-fpm7.2' [22:26:18] RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [22:26:28] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:26:38] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:27:01] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:27:05] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:31:15] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:31:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjCMB [22:31:18] [02miraheze/puppet] 07paladox 0301a79c3 - Include vpncloud role on mw* [22:31:19] [02puppet] 07paladox opened pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:33:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjCMu [22:33:12] [02miraheze/puppet] 07paladox 03ef13fda - Update vpncloud.pp [22:33:14] [02puppet] 07paladox synchronize pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:33:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjCM2 [22:34:00] [02miraheze/puppet] 07paladox 03eddd715 - Update miraheze-internal.net.erb [22:34:02] [02puppet] 07paladox synchronize pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:34:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjCMa [22:34:46] [02miraheze/puppet] 07paladox 032926f52 - Update mw1.yaml [22:34:48] [02puppet] 07paladox synchronize pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:34:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjCMV [22:34:57] [02miraheze/puppet] 07paladox 031a1c05c - Update mw2.yaml [22:34:58] [02puppet] 07paladox synchronize pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:35:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjCMw [22:35:11] [02miraheze/puppet] 07paladox 031a96cd7 - Update mw3.yaml [22:35:13] [02puppet] 07paladox synchronize pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:35:18] [02puppet] 07paladox closed pull request 03#1000: Include vpncloud role on mw* - 13https://git.io/fjCMR [22:35:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fjCMr [22:35:21] [02miraheze/puppet] 07paladox 03cea293f - Include vpncloud role on mw* (#1000) * Include vpncloud role on mw* * Update vpncloud.pp * Update miraheze-internal.net.erb * Update mw1.yaml * Update mw2.yaml * Update mw3.yaml [22:36:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCMi [22:36:31] [02miraheze/puppet] 07paladox 03a79d8ea - Update vpncloud.pp [22:38:55] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:41:35] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCM1 [22:41:36] [02miraheze/dns] 07paladox 03683739a - Depool cp4 [22:42:54] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:44:58] !log updating some packages on cp4 [22:45:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:45:29] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [22:45:42] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [22:45:47] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [22:45:58] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [22:46:15] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [22:47:42] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:47:47] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:47:53] [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fjCM9 [22:47:55] [02miraheze/dns] 07paladox 036da4417 - Revert "Depool cp4" This reverts commit 683739a9ebdcef39a544136186641cc82bc06ce4. [22:47:56] [02miraheze/dns] 07paladox 0370f858e - Depool cp2 [22:53:29] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:53:57] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:54:15] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:56:44] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjCDU [22:56:46] [02miraheze/dns] 07paladox 032d0f713 - Revert "Depool cp2" This reverts commit 70f858e1f38d706f1330c8e51f30c55096fa279c. [22:58:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjCDk [22:58:47] [02miraheze/puppet] 07paladox 03cef74c7 - Add vpncloud role to cp* [22:58:49] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [22:58:50] [02puppet] 07paladox opened pull request 03#1001: Add vpncloud role to cp* - 13https://git.io/fjCDI [23:00:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjCDt [23:00:01] [02miraheze/puppet] 07paladox 03a2e0407 - Update vpncloud.pp [23:00:03] [02puppet] 07paladox synchronize pull request 03#1001: Add vpncloud role to cp* - 13https://git.io/fjCDI [23:00:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjCDq [23:00:38] [02miraheze/puppet] 07paladox 0334f2f56 - Update miraheze-internal.net.erb [23:00:40] [02puppet] 07paladox synchronize pull request 03#1001: Add vpncloud role to cp* - 13https://git.io/fjCDI [23:00:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjCDm [23:00:58] [02miraheze/puppet] 07paladox 03c1d579e - Update cp2.yaml [23:00:59] [02puppet] 07paladox synchronize pull request 03#1001: Add vpncloud role to cp* - 13https://git.io/fjCDI [23:01:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjCDY [23:01:11] [02miraheze/puppet] 07paladox 03641a8e0 - Update cp4.yaml [23:01:13] [02puppet] 07paladox synchronize pull request 03#1001: Add vpncloud role to cp* - 13https://git.io/fjCDI [23:01:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/fjCDO [23:01:27] [02miraheze/puppet] 07paladox 03b26978d - Delete cp5.yaml [23:03:46] [02puppet] 07paladox closed pull request 03#1001: Add vpncloud role to cp* - 13https://git.io/fjCDI [23:03:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fjCDZ [23:03:49] [02miraheze/puppet] 07paladox 03085f4fe - Add vpncloud role to cp* (#1001) * Add vpncloud role to cp* * Update vpncloud.pp * Update miraheze-internal.net.erb * Update cp2.yaml * Update cp4.yaml [23:03:50] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [23:03:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [23:13:47] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:14:41] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:15:10] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:29] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:15:42] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:15:58] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:16:13] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:19:01] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal] [23:21:47] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:22:44] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:23:12] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:23:29] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:42] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:57] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:13] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:24:54] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures