[01:23:14] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5060 MB (20% inode=95%); [06:25:14] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5239 MB (21% inode=95%); [14:57:55] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [15:02:21] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [15:03:55] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:18] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:50:10] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/fjmhW [15:50:12] [02miraheze/MirahezeMagic] 07translatewiki 03b96c929 - Localisation updates from https://translatewiki.net. [15:50:13] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjmhl [15:50:15] [02miraheze/CreateWiki] 07translatewiki 035e0aa46 - Localisation updates from https://translatewiki.net. [16:41:01] 503 [16:41:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [16:42:01] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:09] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [16:42:38] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3858 bytes in 0.010 second response time [16:42:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [16:43:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:43:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:43:40] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [16:43:59] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 23221 bytes in 0.010 second response time [16:44:08] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 59% [16:44:17] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:42] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [16:45:07] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3860 bytes in 0.513 second response time [16:46:01] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3871 bytes in 0.704 second response time [16:46:11] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 23221 bytes in 0.009 second response time [16:46:13] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 23221 bytes in 0.012 second response time [16:46:38] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24051 bytes in 0.027 second response time [16:46:42] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 20% [16:46:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:47:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:47:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:47:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:47:40] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [16:48:01] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24051 bytes in 0.973 second response time [16:48:08] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 7% [16:48:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:50:05] lol [16:59:44] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24052 bytes in 9.734 second response time [17:11:32] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:54] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24086 bytes in 0.502 second response time [18:51:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [18:51:13] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3860 bytes in 0.401 second response time [18:51:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:51:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:52:01] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:17] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [18:52:42] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [18:52:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [18:52:45] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3858 bytes in 0.010 second response time [18:53:40] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [18:54:01] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3871 bytes in 0.702 second response time [18:54:07] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [18:54:42] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 55% [18:55:41] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 43% [18:56:01] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24074 bytes in 1.075 second response time [18:56:07] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 23221 bytes in 0.020 second response time [18:56:15] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 23221 bytes in 0.009 second response time [18:56:38] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24051 bytes in 0.011 second response time [18:56:42] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 1% [18:56:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:57:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:57:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:57:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:57:40] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 16% [18:58:08] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 12% [18:58:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [19:03:21] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24052 bytes in 0.599 second response time [20:31:50] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:45] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24086 bytes in 0.500 second response time [21:51:55] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [21:53:51] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:57:36] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:44] !log verified domain ownership within GitHub [21:57:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:59:31] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24085 bytes in 0.501 second response time [23:49:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYLk [23:49:13] [02miraheze/mw-config] 07paladox 03838d08d - Add TemplateWizard to LocalExtensions - T4286 This was missed when installing it in https://github.com/miraheze/mw-config/commit/ccccf1177974c51cb87c1d421c3c297e4aedcf38 [23:49:13] Title: [ Add TemplateWizard to ManageWiki (#2569) · miraheze/mw-config@ccccf11 · GitHub ] - github.com [23:54:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYLL [23:54:51] [02miraheze/mw-config] 07paladox 033b30ebf - Move wgMaxImageArea under // files section