[00:05:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju2X [00:05:15] [02miraheze/services] 07MirahezeSSLBot 03f433326 - BOT: Updating services config for wikis [00:22:30] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5060 MB (20% inode=94%); [01:15:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuag [01:15:16] [02miraheze/services] 07MirahezeSSLBot 03aa910bb - BOT: Updating services config for wikis [01:36:07] !log installing netcat on test1 [01:36:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:26:30] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5400 MB (22% inode=94%); [08:04:49] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.011 second response time [08:05:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [08:05:21] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [08:05:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [08:05:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [08:06:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [08:06:32] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [08:06:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [08:09:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [08:10:20] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:12:44] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 26% [08:15:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 29% [08:16:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [08:21:41] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [08:24:17] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 9% [08:25:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45% [08:27:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 27% [08:28:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [08:28:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 52% [08:30:44] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 94% [08:31:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [08:37:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 26% [08:40:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 57% [08:41:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 50% [08:42:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [08:43:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 34% [08:43:51] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4227 bytes in 0.031 second response time [08:45:51] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19057 bytes in 0.174 second response time [08:47:41] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 41% [08:49:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [08:50:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 54% [08:52:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [08:53:51] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 3283 bytes in 0.062 second response time [08:56:41] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [08:58:38] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17723 bytes in 0.080 second response time [09:03:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [09:03:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 39% [09:06:26] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [09:07:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [09:07:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [09:08:22] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17723 bytes in 0.079 second response time [09:11:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 32% [09:13:01] paladox: Reception123: 503 [09:15:14] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [09:15:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [09:16:47] PROBLEM - db4 MySQL on db4 is CRITICAL: Too many connections [09:18:41] RECOVERY - db4 MySQL on db4 is OK: Uptime: 732961 Threads: 509 Questions: 700743124 Slow queries: 33378522 Opens: 47681281 Flush tables: 1 Open tables: 800 Queries per second avg: 956.044 [09:20:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 54% [09:22:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [09:24:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 59% [09:26:17] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [09:38:31] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17723 bytes in 0.068 second response time [09:47:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 26% [09:55:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [09:56:05] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [09:59:55] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 348 bytes in 0.013 second response time [10:01:30] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [10:14:47] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41136 bytes in 0.243 second response time [10:14:55] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5806 bytes in 0.043 second response time [10:15:30] RECOVERY - db4 MySQL on db4 is OK: Uptime: 174 Threads: 63 Questions: 10503 Slow queries: 569 Opens: 672 Flush tables: 1 Open tables: 666 Queries per second avg: 60.362 [10:15:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [10:15:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [10:15:51] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17723 bytes in 0.074 second response time [10:15:51] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19072 bytes in 0.145 second response time [10:16:17] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 8% [10:16:32] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [10:16:44] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [10:17:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:17:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:17:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 2% [11:32:07] revi thanks, john seems to have fixed the problem. [11:42:54] Hi! Here is the list of currently open high priority tasks on Phabricator [11:43:01] No updates for 3 days - https://phabricator.miraheze.org/T4438 - Upgrade elasticsearch to version 6 - authored by Southparkfan, assigned to Paladox [11:43:07] No updates for 4 days - https://phabricator.miraheze.org/T4425 - Fix all mysql tables that are using latin rather then utf8mb4 - authored by Paladox, assigned to None [11:43:14] No updates for 3 days - https://phabricator.miraheze.org/T4260 - Migrate all wikis to elasticsearch - authored by Southparkfan, assigned to None [13:08:52] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [13:08:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjuXj [13:08:55] [02miraheze/puppet] 07paladox 0322da303 - Add a check to check for 1% of the disk DB4 has been running out of space lately due to the build up of bin logs files. But because it's gone passed 10% left, we are not notified. Adding another check for 1% left will make sure we will be able to respond quickly before db4 goes down. Bug: T4444 [13:08:57] [02puppet] 07paladox opened pull request 03#1019: Add a check to check for 1% of the disk - 13https://git.io/fju1e [13:09:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fju1f [13:09:51] [02miraheze/puppet] 07paladox 03f1ba3a6 - Update config.pp [13:09:53] [02puppet] 07paladox synchronize pull request 03#1019: Add a check to check for 1% of the disk - 13https://git.io/fju1e [13:10:10] [02puppet] 07paladox synchronize pull request 03#1019: Add a check to check for 1% of the disk - 13https://git.io/fju1e [13:10:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fju1U [13:10:13] [02miraheze/puppet] 07paladox 0350d0138 - Update config.pp [13:10:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju1T [13:10:16] [02miraheze/services] 07MirahezeSSLBot 039ca27ea - BOT: Updating services config for wikis [13:26:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju1z [13:26:13] [02miraheze/puppet] 07paladox 03f54cbbb - Whitelist image.tmdb.org - T4423#84489 [13:53:05] [02puppet] 07JohnFLewis commented on pull request 03#1019: Add a check to check for 1% of the disk - 13https://git.io/fjuMw [13:53:07] [02puppet] 07JohnFLewis closed pull request 03#1019: Add a check to check for 1% of the disk - 13https://git.io/fju1e [13:53:37] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [13:53:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:15:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjuDn [14:15:31] [02miraheze/puppet] 07paladox 03c86adab - Reduce check_disk to check for 5% left of the disk This makes sure to only report a critical if only there's 5% of the disk left or a warning if only 10% is left. [14:15:33] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [14:15:35] [02puppet] 07paladox opened pull request 03#1020: Reduce check_disk to check for 5% left of the disk - 13https://git.io/fjuDc [14:16:53] [02puppet] 07paladox closed pull request 03#1020: Reduce check_disk to check for 5% left of the disk - 13https://git.io/fjuDc [14:16:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuDB [14:16:56] [02miraheze/puppet] 07paladox 033460d78 - Reduce check_disk to check for 5% left of the disk (#1020) This makes sure to only report a critical if only there's 5% of the disk left or a warning if only 10% is left. [14:16:58] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [14:16:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:21:52] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 30866 MB (8% inode=96%); [15:10:51] [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL [15:10:53] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjuSe [15:10:54] [02miraheze/CreateWiki] 07paladox 03cd5fcdd - Return an empty array instead of false in getFormDescriptor This fixes an issue where a request is not found, but instead of returning an empty array, it returned false, which caused an issue in HTMLForm.php. [15:10:56] [02CreateWiki] 07paladox opened pull request 03#115: Return an empty array instead of false in getFormDescriptor - 13https://git.io/fjuSv [15:18:46] SSL certificates expired [15:22:36] paladox: ^^ [15:22:43] oh [15:22:45] * paladox renews [15:25:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSl [15:25:48] [02miraheze/ssl] 07paladox 03fe8e955 - Update drag.lgbt.crt [15:26:17] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [15:26:20] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [15:26:29] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [15:26:54] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [15:27:06] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSB [15:27:08] [02miraheze/ssl] 07paladox 03bf81c7d - Update monarchists.wiki.crt [15:27:37] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [15:27:41] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [15:28:15] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSR [15:28:17] [02miraheze/ssl] 07paladox 03534b71d - Update nonciclopedia.org.crt [15:29:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuS0 [15:29:07] [02miraheze/ssl] 07paladox 0301c7d79 - Update nonsensopedia.org.crt [15:30:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSE [15:30:06] [02miraheze/ssl] 07paladox 03936631f - Update stablestate.org.crt [15:31:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSu [15:31:06] [02miraheze/ssl] 07paladox 03173f1c2 - Update wiki.joust.ro.crt [15:34:25] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuS2 [15:34:26] [02miraheze/ssl] 07paladox 03e16933a - Update wikitranslate.tk.crt [15:35:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSo [15:35:35] [02miraheze/ssl] 07paladox 0337f810a - Update guiasdobrasil.com.br.crt [15:36:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuS6 [15:36:26] [02miraheze/ssl] 07paladox 03c08b969 - Update interno.guiasdobrasil.com.br.crt [15:37:37] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSX [15:37:38] [02miraheze/ssl] 07paladox 03b25fb91 - Update islamkosh.tk.crt [15:38:36] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuS1 [15:38:37] [02miraheze/ssl] 07paladox 038e89345 - Update runzeppelin.ru.crt [15:39:53] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSM [15:39:54] [02miraheze/ssl] 07paladox 03f694b34 - Update wiki.casual4casuals.com.crt [15:40:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSy [15:40:46] [02miraheze/ssl] 07paladox 03d0e7463 - Update wiki.hyperborian.org.crt [15:41:52] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSS [15:41:54] [02miraheze/ssl] 07paladox 032fc59da - Update crimesciencewiki.org.crt [15:42:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSH [15:42:48] [02miraheze/ssl] 07paladox 03d65a367 - Update wiki.coderdojosaintpaul.org.crt [15:43:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuS7 [15:43:45] [02miraheze/ssl] 07paladox 0394559cc - Update wiki.casadocarvalho.net.crt [15:44:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSd [15:44:50] [02miraheze/ssl] 07paladox 03c5d17b2 - Update endlesssea.lucyawrey.com.crt [15:45:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSb [15:45:46] [02miraheze/ssl] 07paladox 0313fd36f - Update wiki.landan.ca.crt [15:46:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSN [15:46:50] [02miraheze/ssl] 07paladox 03de38feb - Update bebaskanpengetahuan.id.crt [15:47:39] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSp [15:47:41] [02miraheze/ssl] 07paladox 0319d923d - Update wiki.ripto.gq.crt [15:48:35] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuSh [15:48:37] [02miraheze/ssl] 07paladox 0330dda93 - Update documentation.aqfer.com.crt [15:49:31] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9v [15:49:32] [02miraheze/ssl] 07paladox 030e7bf34 - Update theatlas.pw.crt [15:50:42] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9J [15:50:43] [02miraheze/ssl] 07paladox 03a0f6d3f - Update knuxwiki.com.crt [15:52:18] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:52:32] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9T [15:52:33] [02miraheze/ssl] 07paladox 0390a1ec7 - Update www.wikimicrofinanza.it.crt [15:52:59] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 03 Sep 2019 02:30:42 PM GMT +0000. [15:53:07] RECOVERY - interno.guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'interno.guiasdobrasil.com.br' will expire on Tue 03 Sep 2019 02:36:06 PM GMT +0000. [15:53:09] RECOVERY - nonsensopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonsensopedia.org' will expire on Tue 03 Sep 2019 02:28:45 PM GMT +0000. [15:53:15] RECOVERY - wiki.landan.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landan.ca' will expire on Tue 03 Sep 2019 02:45:29 PM GMT +0000. [15:53:17] RECOVERY - documentation.aqfer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'documentation.aqfer.com' will expire on Tue 03 Sep 2019 02:48:08 PM GMT +0000. [15:53:31] RECOVERY - drag.lgbt - LetsEncrypt on sslhost is OK: OK - Certificate 'drag.lgbt' will expire on Tue 03 Sep 2019 02:25:23 PM GMT +0000. [15:53:37] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:41] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:53:45] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 03 Sep 2019 02:29:38 PM GMT +0000. [15:53:45] RECOVERY - guiasdobrasil.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'guiasdobrasil.com.br' will expire on Tue 03 Sep 2019 02:35:01 PM GMT +0000. [15:53:49] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Tue 03 Sep 2019 02:27:54 PM GMT +0000. [15:54:00] RECOVERY - wikitranslate.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitranslate.tk' will expire on Tue 03 Sep 2019 02:34:01 PM GMT +0000. [15:54:03] RECOVERY - crimesciencewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crimesciencewiki.org' will expire on Tue 03 Sep 2019 02:41:35 PM GMT +0000. [15:54:04] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Tue 03 Sep 2019 02:46:27 PM GMT +0000. [15:54:06] RECOVERY - wiki.ripto.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ripto.gq' will expire on Tue 03 Sep 2019 02:47:19 PM GMT +0000. [15:54:14] RECOVERY - wiki.casual4casuals.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.casual4casuals.com' will expire on Tue 03 Sep 2019 02:39:13 PM GMT +0000. [15:54:18] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Tue 03 Sep 2019 02:26:48 PM GMT +0000. [15:54:20] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:54:21] RECOVERY - wiki.coderdojosaintpaul.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.coderdojosaintpaul.org' will expire on Tue 03 Sep 2019 02:42:27 PM GMT +0000. [15:54:24] RECOVERY - wiki.casadocarvalho.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.casadocarvalho.net' will expire on Tue 03 Sep 2019 02:43:25 PM GMT +0000. [15:54:27] RECOVERY - knuxwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'knuxwiki.com' will expire on Tue 03 Sep 2019 02:50:01 PM GMT +0000. [15:54:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:54:30] RECOVERY - endlesssea.lucyawrey.com - LetsEncrypt on sslhost is OK: OK - Certificate 'endlesssea.lucyawrey.com' will expire on Tue 03 Sep 2019 02:44:34 PM GMT +0000. [15:54:32] RECOVERY - islamkosh.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'islamkosh.tk' will expire on Tue 03 Sep 2019 02:37:21 PM GMT +0000. [15:54:42] RECOVERY - theatlas.pw - LetsEncrypt on sslhost is OK: OK - Certificate 'theatlas.pw' will expire on Tue 03 Sep 2019 02:49:10 PM GMT +0000. [15:54:46] RECOVERY - wiki.hyperborian.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hyperborian.org' will expire on Tue 03 Sep 2019 02:40:23 PM GMT +0000. [15:54:51] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Tue 03 Sep 2019 02:38:11 PM GMT +0000. [15:54:54] RECOVERY - www.wikimicrofinanza.it - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikimicrofinanza.it' will expire on Tue 03 Sep 2019 02:50:59 PM GMT +0000. [15:54:55] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:56:21] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9Y [15:56:22] [02miraheze/ssl] 07paladox 0360e4289 - Update wiki.ombre.io.crt [15:57:15] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9G [15:57:17] [02miraheze/ssl] 07paladox 03bef5d36 - Update wiki.openhatch.org.crt [15:58:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9Z [15:58:11] [02miraheze/ssl] 07paladox 03bba9ca4 - Update wiki.doverdistrictscouts.co.uk.crt [15:59:14] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9l [15:59:16] [02miraheze/ssl] 07paladox 0307b0df7 - Update wiki.warfra.ml.crt [16:00:11] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju98 [16:00:13] [02miraheze/ssl] 07paladox 03e74e21d - Update wiki.veloren.net.crt [16:01:55] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9R [16:01:57] [02miraheze/ssl] 07paladox 038de592f - Update prfm.wiki.crt [16:03:10] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9E [16:03:11] [02miraheze/ssl] 07paladox 0336cd00a - Update kkutu.wiki.crt [16:04:15] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9g [16:04:16] [02miraheze/ssl] 07paladox 03f298e9a - Update pc-architect.tk.crt [16:05:12] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju92 [16:05:14] [02miraheze/ssl] 07paladox 03c08c066 - Update encyclopediaofastrobiology.org.crt [16:05:39] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:05:52] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:06:15] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9w [16:06:16] [02miraheze/ssl] 07paladox 032fe5c36 - Update electowiki.org.crt [16:06:18] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:06:27] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:06:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:06:46] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fju9r [16:06:48] [02miraheze/ssl] 07paladox 03056fee0 - Update wiki.kybermc.tk.crt [16:06:56] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:08:00] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Tue 03 Sep 2019 03:05:05 PM GMT +0000. [16:08:01] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Tue 03 Sep 2019 03:03:59 PM GMT +0000. [16:08:18] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:08:23] RECOVERY - wiki.ombre.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ombre.io' will expire on Tue 03 Sep 2019 02:55:47 PM GMT +0000. [16:08:24] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [16:08:26] RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Tue 03 Sep 2019 03:01:49 PM GMT +0000. [16:08:37] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:08:41] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:08:42] RECOVERY - wiki.openhatch.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Tue 03 Sep 2019 02:56:55 PM GMT +0000. [16:08:45] RECOVERY - wiki.kybermc.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kybermc.tk' will expire on Tue 03 Sep 2019 03:05:54 PM GMT +0000. [16:08:49] RECOVERY - pc-architect.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'pc-architect.tk' will expire on Tue 03 Sep 2019 03:03:01 PM GMT +0000. [16:08:52] RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Tue 03 Sep 2019 02:59:42 PM GMT +0000. [16:09:25] RECOVERY - wiki.doverdistrictscouts.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.doverdistrictscouts.co.uk' will expire on Tue 03 Sep 2019 02:57:52 PM GMT +0000. [16:09:34] RECOVERY - prfm.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'prfm.wiki' will expire on Tue 03 Sep 2019 03:00:45 PM GMT +0000. [16:09:34] RECOVERY - wiki.warfra.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warfra.ml' will expire on Tue 03 Sep 2019 02:58:52 PM GMT +0000. [16:09:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:09:49] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:11:57] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fju91 [16:11:59] [02miraheze/ssl] 07paladox 03d2c0876 - Remove absurdopedia.wiki ssl certificate Domain no longer pointing at us. [16:12:00] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:12:02] [02ssl] 07paladox opened pull request 03#190: Remove absurdopedia.wiki ssl certificate - 13https://git.io/fju9M [16:12:42] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fju99 [16:12:43] [02miraheze/ssl] 07paladox 03e8bdd3c - Delete absurdopedia.wiki.crt [16:12:45] [02ssl] 07paladox synchronize pull request 03#190: Remove absurdopedia.wiki ssl certificate - 13https://git.io/fju9M [16:12:52] [02ssl] 07paladox closed pull request 03#190: Remove absurdopedia.wiki ssl certificate - 13https://git.io/fju9M [16:12:54] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fju97 [16:12:55] [02miraheze/ssl] 07paladox 033ff8a7a - Remove absurdopedia.wiki ssl certificate (#190) * Remove absurdopedia.wiki ssl certificate Domain no longer pointing at us. * Delete absurdopedia.wiki.crt [16:12:57] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:12:58] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [16:15:12] Hello MillieJonasen! If you have any questions feel free to ask and someone should answer soon. [16:15:14] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[absurdopedia.wiki_private] [16:15:29] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[absurdopedia.wiki_private] [16:24:28] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [16:25:05] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:50:39] !log reset /srv/mediawiki/config on test1 (merge conflicts) [16:50:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:58:17] [02CreateWiki] 07paladox closed pull request 03#115: Return an empty array instead of false in getFormDescriptor - 13https://git.io/fjuSv [16:58:19] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuH2 [16:58:20] [02miraheze/CreateWiki] 07paladox 030bf2577 - Return an empty array instead of false in getFormDescriptor (#115) This fixes an issue where a request is not found, but instead of returning an empty array, it returned false, which caused an issue in HTMLForm.php. [16:58:22] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3 [16:58:23] [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL [16:59:28] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjuHa [16:59:30] [02miraheze/mediawiki] 07paladox 0373cf705 - Update CW [17:13:14] Paladox i have to complain i used to be able to use the sidebar managewiki links to take me to a special page to enter a dbname instead it seems to default to metawiki’s settings 😦 [17:13:31] yeh, you have to go to the wiki now [17:13:38] @Zppix blame JohnLewis :D [17:13:57] Paladox but since wiki manager isnt global that wont work ? [17:14:06] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:15:14] @Zppix john did notifiy the community on the discussion forum. [17:15:24] (was open for 2 weeks) [17:15:35] So whats the point of wiki manager im lost? [17:16:51] i think john's planning on getting rid of wiki managers. [17:29:40] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns [17:29:41] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjuHb [17:29:43] [02miraheze/ManageWiki] 07paladox 03f3c3cd5 - Fix 'autopromote' default value in ::groupPermissions to be null [17:29:51] [02ManageWiki] 07paladox opened pull request 03#105: Fix 'autopromote' default value in ::groupPermissions to be null - 13https://git.io/fjuHN [17:31:47] [02ManageWiki] 07paladox closed pull request 03#105: Fix 'autopromote' default value in ::groupPermissions to be null - 13https://git.io/fjuHN [17:31:48] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuHx [17:31:50] [02miraheze/ManageWiki] 07paladox 037b7de39 - Fix 'autopromote' default value in ::groupPermissions to be null (#105) [17:31:51] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [17:31:53] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [17:32:32] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjuHp [17:32:34] [02miraheze/mediawiki] 07paladox 038662820 - Update MW [17:40:08] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:48:06] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:19] !log sudo apt-get install devscripts fakeroot on test1 [19:13:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:02:30] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:02:35] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:02:39] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:02:42] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:02:58] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:02:59] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:03:04] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:03:11] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:03:18] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 2 minutes ago with 18 failures. Failed resources (up to 3 shown) [21:03:23] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:03:40] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:03:41] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:03:56] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:04:07] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:04:09] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:04:10] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:04:20] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:04:22] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 205 failures. Last run 3 minutes ago with 205 failures. Failed resources (up to 3 shown) [21:04:28] PROBLEM - lizardfs3 Puppet on lizardfs3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:06:41] haha daily puppet failure [21:08:55] Voidwalker: regular like clockwork [21:12:56] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:12:59] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [21:13:07] now here comes the recovery :D [21:13:18] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [21:13:41] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:13:56] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [21:14:06] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:14:09] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:15] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [21:14:27] RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:30] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:36] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:42] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:04] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [21:15:12] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:23] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:40] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:22:44] Paladox, JohnLewis: How do I set up an oauth app for quircwiki [21:24:14] RF1dle: https://meta.miraheze.org/wiki/Special:OAuthConsumerRegistration/propose and ask us to review it once submitted :) [21:24:15] [ Permission error - Miraheze Meta ] - meta.miraheze.org [21:24:28] * JohnLewis hopes that link will work for you [21:25:04] JohnLewis: Can you confirm me for it? [21:25:16] not autoconfirmed on meta [21:25:43] what's your username again on wiki? [21:26:03] JohnLewis: RhinosF1 [21:26:49] RF1dle: try now :) [21:32:20] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [21:34:18] RF1dle: approved [21:38:23] thanks JohnLewis [21:49:05] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [21:51:03] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2% [22:54:44] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjuFb [22:54:46] [02miraheze/CreateWiki] 07JohnFLewis 03931d093 - base for multi db server support [22:58:10] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjuFx [22:58:12] [02miraheze/CreateWiki] 07JohnFLewis 03fea174a - base for multi db server support [23:42:06] PROBLEM - test1 Disk Space on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:14] PROBLEM - test1 SSH on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:42:22] PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:23] PROBLEM - Host test1 is DOWN: PING CRITICAL - Packet loss = 100% [23:44:00] 10/10 to paladox [23:44:11] oh [23:44:15] it's not really down [23:44:21] i just broke the network somehow.. [23:44:35] RECOVERY - Host test1 is UP: PING OK - Packet loss = 73%, RTA = 0.44 ms [23:44:38] PROBLEM - test1 php-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:38] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:56] RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 5 processes with command name 'php-fpm7.3' [23:45:01] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.024 second response time [23:46:11] RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 9719 MB (23% inode=97%); [23:46:21] RECOVERY - test1 SSH on test1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [23:46:26] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.01, 0.03, 0.04 [23:53:24] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:31] PROBLEM - test1 Disk Space on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:42] PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:54:43] PROBLEM - test1 SSH on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:45] PROBLEM - Host test1 is DOWN: PING CRITICAL - Packet loss = 100%