[00:17:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAef [00:17:11] [02miraheze/puppet] 07paladox 0395f4619 - citoid: Fix citoid group gid [00:22:45] PROBLEM - misc3 Puppet on misc3 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [00:24:42] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 29 seconds ago with 5 failures. Failed resources (up to 3 shown): Package[libgconf2-4],Exec[zotero_npm],Exec[citoid_npm],Exec[git_clone_restbase] [00:24:51] PROBLEM - misc3 citoid on misc3 is CRITICAL: connect to address 185.52.1.71 and port 6927: Connection refused [00:25:00] PROBLEM - misc3 proton on misc3 is CRITICAL: connect to address 185.52.1.71 and port 3000: Connection refused [00:25:09] PROBLEM - misc3 zotero on misc3 is CRITICAL: connect to address 185.52.1.71 and port 1969: Connection refused [00:25:13] PROBLEM - misc3 electron on misc3 is CRITICAL: connect to address 185.52.1.71 and port 3000: Connection refused [00:25:50] PROBLEM - misc3 restbase on misc3 is CRITICAL: connect to address 185.52.1.71 and port 7231: Connection refused [00:27:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAeq [00:27:12] [02miraheze/puppet] 07paladox 03d536e04 - restbase: Checkout v0.19.3 [00:32:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAeO [00:32:20] [02miraheze/puppet] 07paladox 035e38bfe - electron: Only install libgconf2-4 on stretch libgconf2-4 is replaced with gconf-gsettings-backend under buster. [00:36:18] PROBLEM - misc3 lizard.miraheze.org HTTPS on misc3 is CRITICAL: connect to address 185.52.1.71 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:41:59] RECOVERY - misc3 lizard.miraheze.org HTTPS on misc3 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 381 bytes in 0.019 second response time [00:42:11] RECOVERY - misc3 proton on misc3 is OK: TCP OK - 1.001 second response time on 185.52.1.71 port 3000 [00:42:39] RECOVERY - misc3 citoid on misc3 is OK: TCP OK - 0.003 second response time on 185.52.1.71 port 6927 [00:42:48] RECOVERY - misc3 zotero on misc3 is OK: TCP OK - 0.002 second response time on 185.52.1.71 port 1969 [00:43:13] RECOVERY - misc3 electron on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.71 port 3000 [00:43:15] RECOVERY - misc3 restbase on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.71 port 7231 [00:47:12] !log sudo ufw allow from 185.52.1.71 on misc3 [00:47:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:48:15] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 3 failures [00:50:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAeR [00:50:57] [02miraheze/puppet] 07paladox 03254ceb8 - salt: Only install saltstack apt repo if on stretch only [00:53:12] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:53:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAez [00:53:15] [02miraheze/puppet] 07paladox 0304b5c03 - lizardfs: Replace master with 185.52.1.71 [00:53:17] [02puppet] 07paladox opened pull request 03#1070: lizardfs: Replace master with 185.52.1.71 - 13https://git.io/fjAeg [00:54:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAe2 [00:54:27] [02miraheze/puppet] 07paladox 030dbd510 - Update master.pp [00:54:28] [02puppet] 07paladox synchronize pull request 03#1070: lizardfs: Replace master with 185.52.1.71 - 13https://git.io/fjAeg [00:54:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAeV [00:54:40] [02miraheze/puppet] 07paladox 0313e65b0 - Update storage.pp [00:54:42] [02puppet] 07paladox synchronize pull request 03#1070: lizardfs: Replace master with 185.52.1.71 - 13https://git.io/fjAeg [00:55:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAew [00:55:03] [02miraheze/puppet] 07paladox 030d49412 - Update client.pp [00:55:04] [02puppet] 07paladox synchronize pull request 03#1070: lizardfs: Replace master with 185.52.1.71 - 13https://git.io/fjAeg [00:55:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAer [00:55:35] [02miraheze/puppet] 07paladox 03c958cc5 - Update common.yaml [00:55:36] [02puppet] 07paladox synchronize pull request 03#1070: lizardfs: Replace master with 185.52.1.71 - 13https://git.io/fjAeg [00:57:23] [02puppet] 07paladox closed pull request 03#1070: lizardfs: Replace master with 185.52.1.71 - 13https://git.io/fjAeg [00:57:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fjAeo [00:57:26] [02miraheze/puppet] 07paladox 033a768dc - lizardfs: Replace master with 185.52.1.71 (#1070) * lizardfs: Replace master with 185.52.1.71 * Update master.pp * Update storage.pp * Update client.pp * Update common.yaml [00:58:01] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:58:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAeK [00:58:16] [02miraheze/puppet] 07paladox 032503dc0 - Update minion.pp [00:58:26] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [00:58:34] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [00:58:55] PROBLEM - lizardfs1 Puppet on lizardfs1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [00:59:58] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [01:00:13] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [01:00:53] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [01:03:51] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 1 minute ago with 4 failures. Failed resources (up to 3 shown): Service[lizardfs-master],Service[lizardfs-cgiserv],Package[apache2],Service[salt-minion] [01:06:25] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static],Mount[/mnt/mediawiki-trash] [01:06:33] !log added 1gb of swap to misc3 (new host) [01:06:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:09:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAeH [01:09:41] [02miraheze/puppet] 07paladox 035e38349 - Update misc3.yaml [01:10:37] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:10:54] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:10:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:11:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:11:09] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:14] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [01:11:19] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:11:33] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:34] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:39] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 4 failures [01:11:41] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:44] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:11:49] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [01:11:56] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:56] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 54% [01:11:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:12:02] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:22] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:12:39] known [01:12:53] i dunno why lizardfs on misc3 (old) failed.. [01:13:08] fixed [01:13:20] ah [01:13:30] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.393 second response time [01:13:31] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.392 second response time [01:13:32] appears it OOM'd? Or at least stopped on it's own [01:13:41] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.670 second response time [01:13:50] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.003 second response time [01:13:57] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.005 second response time [01:14:18] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.697 second response time [01:14:40] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.004 second response time [01:14:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:15:04] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 1.441 second response time [01:15:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:15:14] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 1% [01:15:18] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.395 second response time [01:15:19] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:15:35] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 1 minute ago with 4 failures. Failed resources (up to 3 shown): Service[lizardfs-master],Service[lizardfs-cgiserv],Package[apache2],Service[salt-minion] [01:15:44] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:15:49] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [01:15:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:15:56] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 2% [01:17:36] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 0 failures [01:19:09] PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: connect to address 185.52.1.71 and port 9419: Connection refused [01:19:34] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:21:08] RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.003 second response time on 185.52.1.71 port 9419 [01:22:38] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [01:23:00] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:25:33] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [01:28:32] !log restarting lizardfs chunkserver on lizardfs[123] to pickup a master ip change [01:28:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:29:27] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [01:31:38] !log reboot mw1 [01:32:22] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:23] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:34] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:40] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:33:02] known [01:33:09] i need to switch the clients on mw* [01:33:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:33:11] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:18] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:33:30] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:31] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:35] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:33:44] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:33:46] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:49] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 55% [01:33:55] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:56] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:01] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:04] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:21] PROBLEM - mw1 SSH on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:03] PROBLEM - mw1 php-fpm on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:35:09] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.010 second response time [01:35:14] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [01:35:22] PROBLEM - Host mw1 is DOWN: PING CRITICAL - Packet loss = 100% [01:35:33] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.393 second response time [01:35:46] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 0.393 second response time [01:35:49] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 29% [01:35:50] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.256 second response time [01:36:04] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.850 second response time [01:36:24] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.737 second response time [01:36:34] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [01:36:40] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.007 second response time [01:36:42] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.214 second response time [01:36:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:37:14] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 5% [01:37:22] RECOVERY - Host mw1 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [01:37:27] PROBLEM - mw1 Disk Space on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:37:27] PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:27] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:37:42] RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 25967 MB (33% inode=98%); [01:37:49] RECOVERY - mw1 MirahezeRenewSsl on mw1 is OK: TCP OK - 0.001 second response time on 185.52.1.75 port 5000 [01:38:00] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 0.45, 0.12, 0.04 [01:38:00] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.009 second response time [01:38:02] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.020 second response time [01:38:11] RECOVERY - mw1 SSH on mw1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [01:38:18] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [01:38:52] RECOVERY - mw1 php-fpm on mw1 is OK: PROCS OK: 12 processes with command name 'php-fpm7.2' [01:39:30] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 0.393 second response time [01:39:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:39:39] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.826 second response time [01:39:44] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [01:40:26] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [01:41:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [01:41:18] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [01:43:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvU [01:44:00] [02miraheze/dns] 07paladox 03fc9b8d3 - switch misc3 ip [01:44:16] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [01:44:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjAvT [01:44:19] [02miraheze/puppet] 07paladox 034194398 - switch misc3 ip [01:44:20] [02puppet] 07paladox opened pull request 03#1071: switch misc3 ip - 13https://git.io/fjAvk [01:44:33] [02puppet] 07paladox closed pull request 03#1071: switch misc3 ip - 13https://git.io/fjAvk [01:44:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvI [01:44:36] [02miraheze/puppet] 07paladox 031df4ed4 - switch misc3 ip (#1071) [01:45:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvL [01:45:02] [02miraheze/puppet] 07paladox 0394ad183 - vpncloud: Remove 185.52.1.144 [01:45:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [01:45:15] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [01:54:03] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 401 Unauthorized [01:54:05] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 401 Unauthorized [01:54:34] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 401 Unauthorized [01:55:07] PROBLEM - cp4 Stunnel Http for misc3 on cp4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 401 Unauthorized [01:55:08] PROBLEM - cp3 Stunnel Http for misc3 on cp3 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 401 Unauthorized [01:55:19] PROBLEM - cp2 Stunnel Http for misc3 on cp2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8203: HTTP/1.1 401 Unauthorized [01:56:56] hmm [02:13:34] I found that pages with more than 250 revisions cannot deleted [02:13:53] at least by local admin [02:14:09] Then who have the right to delete those pages? [02:15:47] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvg [02:15:49] [02miraheze/dns] 07paladox 034541dd5 - add lizardfs4 to dns [02:16:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjAva [02:16:56] [02miraheze/puppet] 07paladox 03197f789 - Add lizardfs4 [02:16:57] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:16:59] [02puppet] 07paladox opened pull request 03#1072: Add lizardfs4 - 13https://git.io/fjAvV [02:17:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/fjAvw [02:17:33] [02miraheze/puppet] 07paladox 0392e098b - Create lizardfs4.yaml [02:17:34] [02puppet] 07paladox synchronize pull request 03#1072: Add lizardfs4 - 13https://git.io/fjAvV [02:18:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjAvr [02:18:02] [02miraheze/puppet] 07paladox 039e9f1d9 - Update storage_firewall.yaml [02:18:04] [02puppet] 07paladox synchronize pull request 03#1072: Add lizardfs4 - 13https://git.io/fjAvV [02:18:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjAvo [02:18:19] [02miraheze/puppet] 07paladox 037331629 - Update config.yaml [02:18:20] [02puppet] 07paladox synchronize pull request 03#1072: Add lizardfs4 - 13https://git.io/fjAvV [02:18:23] PROBLEM - lizardfs4 Puppet on lizardfs4 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [02:18:38] [02puppet] 07paladox closed pull request 03#1072: Add lizardfs4 - 13https://git.io/fjAvV [02:18:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±3] 13https://git.io/fjAvK [02:18:41] [02miraheze/puppet] 07paladox 03c193e10 - Add lizardfs4 (#1072) * Add lizardfs4 * Create lizardfs4.yaml * Update storage_firewall.yaml * Update config.yaml [02:18:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:18:44] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:22:18] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 seconds ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [02:32:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvy [02:32:53] [02miraheze/puppet] 07paladox 033f0e057 - lizardfs1: Mark for removal [02:33:56] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:34:03] !log root@lizardfs1:/home/paladox# sudo service lizardfs-chunkserver restart [02:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:35:27] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvS [02:35:29] [02miraheze/dns] 07paladox 0316a9aca - Add lizardfs5 to dns [02:40:00] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:40:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/fjAvH [02:40:03] [02miraheze/puppet] 07paladox 0329be022 - add lizardfs5 [02:40:04] [02puppet] 07paladox opened pull request 03#1073: add lizardfs5 - 13https://git.io/fjAvQ [02:40:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjAv5 [02:40:50] [02miraheze/puppet] 07paladox 0310163fd - Update storage_firewall.yaml [02:40:51] [02puppet] 07paladox synchronize pull request 03#1073: add lizardfs5 - 13https://git.io/fjAvQ [02:41:29] [02puppet] 07paladox synchronize pull request 03#1073: add lizardfs5 - 13https://git.io/fjAvQ [02:41:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjAvF [02:41:32] [02miraheze/puppet] 07paladox 031931f5f - Update storage_firewall.yaml [02:43:33] [02puppet] 07paladox closed pull request 03#1073: add lizardfs5 - 13https://git.io/fjAvQ [02:43:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjAvA [02:43:36] [02miraheze/puppet] 07paladox 0302eab2c - add lizardfs5 (#1073) * add lizardfs5 * Update storage_firewall.yaml * Update storage_firewall.yaml [02:43:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:43:39] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [02:43:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvx [02:43:52] [02miraheze/puppet] 07paladox 0347a2eff - Update site.pp [02:44:01] PROBLEM - lizardfs5 Puppet on lizardfs5 is UNKNOWN: NRPE: Unable to read output [02:46:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAvh [02:46:23] [02miraheze/puppet] 07paladox 0319161f6 - Update lizardfs5.yaml [02:46:49] PROBLEM - lizardfs3 Disk Space on lizardfs3 is CRITICAL: DISK CRITICAL - free space: / 9210 MB (5% inode=98%); [02:48:10] PROBLEM - lizardfs2 Disk Space on lizardfs2 is CRITICAL: DISK CRITICAL - free space: / 9207 MB (5% inode=98%); [02:49:57] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:51:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAff [02:51:39] [02miraheze/puppet] 07paladox 035b917bb - lizardfs2: Mark for removal [02:51:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAfJ [02:52:00] [02miraheze/puppet] 07paladox 038dec20e - lizardfs3: Mark for removal [02:54:07] !log root@lizardfs2:/home/paladox# sudo service lizardfs-chunkserver restart [02:54:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:55:15] !log root@lizardfs3:/home/paladox# sudo service lizardfs-chunkserver restart [02:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:05:47] PROBLEM - lizardfs2 Disk Space on lizardfs2 is WARNING: DISK WARNING - free space: / 9224 MB (6% inode=98%); [03:07:05] PROBLEM - lizardfs1 Disk Space on lizardfs1 is WARNING: DISK WARNING - free space: / 9220 MB (6% inode=98%); [03:12:29] PROBLEM - lizardfs3 Disk Space on lizardfs3 is WARNING: DISK WARNING - free space: / 9216 MB (6% inode=98%); [03:43:01] Hello Rich! If you have any questions feel free to ask and someone should answer soon. [03:48:58] hello, I'm the admin for gauntletcomics.miraheze.org and I'm not sure how to change a setting. I've looked at Special:ManageWiki and tried to find it on my own, but failed. [03:50:14] How do I change the icon in the upper left-hand of each page to display an image of my own choosing? Right now it's the Miraheze default of a brown hexagon surrounded by yellow hexagons, but I want to use the Gauntlet Comics logo there. [03:50:46] I don't know what that icon thingie is called, which might be why I've failed in my attempt to figure out how to change it. Thanks for any help. [06:26:42] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2971 MB (12% inode=94%); [07:14:33] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:15] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:43] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:16:31] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4137 files, 8.488MB, 2019-08-11 00:05:00 (2.0 weeks ago) [07:17:14] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Full, 764611 files, 70.58GB, 2019-08-04 02:46:00 (3.0 weeks ago) [07:17:37] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.02, 0.05, 0.06 [08:04:36] PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Full, 4580627 files, 388.1GB, 2019-08-10 08:02:00 (2.1 weeks ago) [12:25:10] RECOVERY - lizardfs3 Disk Space on lizardfs3 is OK: DISK OK - free space: / 16921 MB (11% inode=98%); [12:50:57] Hello orklord! If you have any questions feel free to ask and someone should answer soon. [12:52:36] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Tue 10 Sep 2019 12:50:26 PM GMT +0000). [12:52:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAI9 [12:52:52] [02miraheze/ssl] 07MirahezeSSLBot 030c88133 - Bot: Update SSL cert for guia.cineastas.pt [12:55:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAI5 [12:55:21] [02miraheze/puppet] 07paladox 0314d17da - nrpe: Fix wrong port in check_stunnel_misc2 [12:55:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAId [12:55:44] [02miraheze/puppet] 07paladox 03e49b554 - nrpe: Fix check_stunnel_misc3 to expect 403 not 502 [12:55:50] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is WARNING: WARNING: Full, 80402 files, 2.603GB, 2019-08-10 12:53:00 (2.1 weeks ago) [12:56:12] paladox: any chance you could deploy my mw-config pr once you've done [13:00:36] [02mw-config] 07paladox reviewed pull request 03#2738 commit - 13https://git.io/fjAIx [13:01:06] [02mw-config] 07paladox reviewed pull request 03#2738 commit - 13https://git.io/fjAIp [13:01:23] [02mw-config] 07paladox reviewed pull request 03#2738 commit - 13https://git.io/fjAIh [13:01:39] [02mw-config] 07paladox reviewed pull request 03#2738 commit - 13https://git.io/fjAIj [13:02:14] RhinosF1 ^ [13:02:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjALe [13:02:50] [02miraheze/puppet] 07paladox 03225e286 - nrpe: Fix check_stunnel_misc3 to expect 401 not 403 [13:04:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjALf [13:04:18] [02miraheze/puppet] 07paladox 0348d6c52 - nrpe: Remove expect from check_stunnel_misc2 [13:04:36] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Sat 23 Nov 2019 11:52:44 AM GMT +0000. [13:04:53] RECOVERY - cp4 Stunnel Http for misc3 on cp4 is OK: HTTP OK: Status line output matched "401" - 381 bytes in 0.015 second response time [13:05:16] [02mw-config] 07RhinosF1 synchronize pull request 03#2738: add 4 exts to test1 - 13https://git.io/fjNpK [13:05:51] Paladox: ^ [13:05:51] RECOVERY - cp3 Stunnel Http for misc3 on cp3 is OK: HTTP OK: Status line output matched "401" - 381 bytes in 0.479 second response time [13:05:53] RECOVERY - cp2 Stunnel Http for misc3 on cp2 is OK: HTTP OK: Status line output matched "401" - 381 bytes in 0.300 second response time [13:05:59] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.577 second response time [13:06:04] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.924 second response time [13:06:10] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 41802 bytes in 0.075 second response time [13:06:42] [02mw-config] 07paladox closed pull request 03#2738: add 4 exts to test1 - 13https://git.io/fjNpK [13:06:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fjALk [13:06:45] [02miraheze/mw-config] 07RhinosF1 03fc44fe3 - add 4 exts to test1 (#2738) * add 4 extensions per mediawiki repo * Update LocalSettings.php * add to MW * add 4 to ext-list * enable on test1wiki * Update ManageWikiExtensions.php Lowercase per Paladox [13:07:47] [79c71049890b5ee2e4d5d4a5] [no req] Exception from line 462 of /srv/mediawiki/w/includes/registration/ExtensionProcessor.php: It was attempted to load Disambiguator twice, from /srv/mediawiki/w/extensions/Disambiguator/extension.json and /srv/mediawiki/w/extensions/SoftRedirector/extension.json. [13:08:56] paladox: mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki [13:09:15] that's what i ran [13:09:57] aha! [13:10:03] the wrong submodule was added [13:10:26] paladox: ? [13:10:44] i see Disambiguator in SoftRedirector dir [13:11:28] by that i mean in SoftRedirector extension.json, i see Disambiguator [13:11:54] paladox: I'm on it [13:12:04] I'll have a look and chat with MJL [13:12:22] Ah [13:12:28] was the extension renamed RhinosF1 ? [13:12:29] https://github.com/wikimedia/mediawiki-extensions-SoftRedirector/tree/REL1_33 [13:12:30] [ GitHub - wikimedia/mediawiki-extensions-SoftRedirector at REL1_33 ] - github.com [13:13:08] It's based off disambiguator [13:13:57] ok, we won't be able to use the extension till 1.34 then [13:14:07] Why? [13:14:16] as i'm not sure if we use the master branch, that they will introduce a breaking change [13:14:51] I've spoke to MJL so hopefully he'll fix it [13:15:21] Unless he backports his changes to REL1_33, we carn't use this extension till 1.34 [13:15:30] since the extension is now conflicting with Disambiguator [13:15:58] I'll see if he can backport and for now we'll have to turn off on test1 [13:16:05] ok [13:16:55] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [13:16:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjALZ [13:16:58] [02miraheze/mw-config] 07paladox 036443af6 - Temporarily disable SoftRedirector [13:16:59] [02mw-config] 07paladox opened pull request 03#2739: Temporarily disable SoftRedirector - 13https://git.io/fjALn [13:17:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjALc [13:17:21] [02miraheze/mw-config] 07paladox 03b1cdb22 - Update LocalSettings.php [13:17:22] [02mw-config] 07paladox synchronize pull request 03#2739: Temporarily disable SoftRedirector - 13https://git.io/fjALn [13:19:38] [02mw-config] 07paladox closed pull request 03#2739: Temporarily disable SoftRedirector - 13https://git.io/fjALn [13:19:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjALC [13:19:41] [02miraheze/mw-config] 07paladox 0349c05fb - Temporarily disable SoftRedirector (#2739) * Temporarily disable SoftRedirector * Update LocalSettings.php [13:20:14] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [13:20:16] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [13:20:37] paladox: now PageDisqus is failing [13:20:45] hmm [13:20:59] PROBLEM - bacula1 Bacula Databases db5 on bacula1 is WARNING: WARNING: Full, 422 files, 19.13GB, 2019-08-10 13:18:00 (2.1 weeks ago) [13:21:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAL4 [13:21:12] [02miraheze/mw-config] 07paladox 03ec20fe5 - Update extension-list [13:22:04] paladox: I think we need some config adding for the disqus shortname [13:22:16] https://www.mediawiki.org/wiki/Extension:PageDisqus [13:22:16] [ Extension:PageDisqus - MediaWiki ] - www.mediawiki.org [13:23:32] Yep we do paladox [13:23:43] That's dangerous that if they miss it out [13:23:58] Ok [13:24:50] paladox: do you want me to try and get a shortname? [13:25:10] Should we use the site name for that? [13:25:37] paladox: you need to get it from Disqus I think [13:25:45] oh [13:26:14] Yeah, that's very dangerous if enabled through ManageWiki without that [13:26:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjALu [13:26:24] [02miraheze/mw-config] 07paladox 039217afd - Disable wmgUsePageDisqus on test1 [13:32:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjALX [13:32:14] [02miraheze/mw-config] 07paladox 03754109d - Add wgPageDisqusShortname to ManageWiki [13:32:15] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [13:32:17] [02mw-config] 07paladox opened pull request 03#2740: Add wgPageDisqusShortname to ManageWiki - 13https://git.io/fjALM [13:33:19] paladox: use test1miraheze as the shortname for test1 [13:33:49] So we could do wgPageDisqusShortname = $wgDBname? [13:34:08] paladox: no I just registered it [13:34:15] oh ok [13:35:31] [02mw-config] 07paladox synchronize pull request 03#2740: Add wgPageDisqusShortname to ManageWiki - 13https://git.io/fjALM [13:35:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjAL7 [13:35:34] [02miraheze/mw-config] 07paladox 034f733b9 - Update ManageWikiSettings.php [13:36:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjALd [13:36:55] [02miraheze/mw-config] 07paladox 03e6eb692 - Update LocalSettings.php [13:36:57] [02mw-config] 07paladox synchronize pull request 03#2740: Add wgPageDisqusShortname to ManageWiki - 13https://git.io/fjALM [13:38:25] [02mw-config] 07paladox closed pull request 03#2740: Add wgPageDisqusShortname to ManageWiki - 13https://git.io/fjALM [13:38:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjALN [13:38:28] [02miraheze/mw-config] 07paladox 03edcc7e1 - Add wgPageDisqusShortname to ManageWiki (#2740) * Add wgPageDisqusShortname to ManageWiki * Update ManageWikiSettings.php * Update LocalSettings.php [13:38:29] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [13:38:31] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [13:39:40] RhinosF1 done [13:40:22] paladox: have you enabled the extension? [13:40:29] yup [13:41:33] paladox: found it, doesn't work on Main_Page by default [13:41:44] it's working though? [13:41:53] paladox: still testing [13:42:37] paladox: can you see my comment? https://test1.miraheze.org/wiki/PageDisqus [13:42:38] [ PageDisqus - Test1 ] - test1.miraheze.org [13:42:49] yup [13:42:55] paladox: then yes [13:43:03] it works!! [13:44:57] !log running lc on mw* [13:45:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:45:50] paladox: +1 for DisqusTag as well [13:46:07] It works for DisqusTag? [13:46:36] paladox: DisqusTag works but needs the same setting as PageDisqus doing [13:46:47] so not sure how that acts without it [13:47:01] ok [13:47:16] paladox: maybe turn off PageDisqus and blank that setting so we can test that first [13:48:22] ok [13:49:22] paladox: if you've just blanked that setting as well then I'm happy with DisqusTag unrestricted [13:50:02] not sure how I can test StopForumSpam yet [13:50:10] Disqus should stay restricted [13:50:23] otherwise users will have issues due to the extension needing the setting. [13:50:53] paladox: PageDisqus should but DisqusTag doesn't fall apart as bad it just opens a blank box [13:51:00] when the link is clicked [13:51:08] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.57, 6.50, 4.32 [13:54:21] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [13:54:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjAtY [13:54:24] [02miraheze/mw-config] 07paladox 03357e8cf - Add wgDisqusShortname to ManageWiki [13:54:27] anways the extension won't work [13:54:31] it's using wrong config names [13:54:40] [02mw-config] 07paladox opened pull request 03#2741: Add wgDisqusShortname to ManageWiki - 13https://git.io/fjAtG [13:55:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fjAtC [13:55:45] [02miraheze/mw-config] 07paladox 03d8a8a95 - Update LocalSettings.php [13:55:47] [02mw-config] 07paladox synchronize pull request 03#2741: Add wgDisqusShortname to ManageWiki - 13https://git.io/fjAtG [13:59:34] paladox: how can we test StopForumSpam [13:59:45] I'm not sure [14:00:35] paladox: neither am I so as it stands the 2 disqus ones are restricted extensions and SoftRedirector is on hold pending me speaking to MJL [14:01:13] yeh [14:01:22] one of those disqus extensions won't work [14:01:29] fixed with https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DisqusTag/+/532242/ [14:01:31] [ Gerrit Code Review ] - gerrit.wikimedia.org [14:01:51] paladox: ok [14:02:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:02:09] but I had DisqusTag working [14:02:16] !log deleted and dropped actualitywiki adullamwiki; aeriontechwiki; ag7732wiki; [14:02:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:02:26] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [14:02:42] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=94%); [14:04:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:04:20] paladox: can you also check the exception log for a blog issue around 14:56 [14:04:26] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [14:04:43] yeh [14:07:11] JohnLewis: any clue how to test StopForumSpam [14:07:49] be banned by SFS and see if you get blocked here, simples [14:08:36] JohnLewis: yes but that means being banned by them which I doubt is going to happen, I mean so we can push it to manage wiki for people to enable [14:08:55] probs [14:09:20] JohnLewis: so can we make a call on whether to pass it for all wikis or? [14:09:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAtE [14:09:33] [02miraheze/mw-config] 07paladox 03348b3cb - blogpage requires PollNY [14:10:35] huh? [14:11:14] JohnLewis: we're supposed to test before letting wikis use them, we can't do that (although it's not broke anything AFAIK) [14:11:15] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.94, 7.50, 7.23 [14:11:44] whoever tests it makes the call if it works [14:12:36] paladox: do you want to explain the issues on https://phabricator.miraheze.org/T4235 and why it's restricted only [14:12:37] [ ⚓ T4235 Enable PageDisqus on cloudytheology.miraheze.org ] - phabricator.miraheze.org [14:13:14] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 11.24, 8.57, 7.63 [14:13:16] done [14:13:33] JohnLewis: well out of 4 extensions, this ain't going well, 2 are restricted due to issues, 1 is removed due to a fatal and one none of us can test [14:14:11] what's the one with fatals? [14:14:26] also if none are added, why has one had config added to ManageWiki? [14:14:37] JohnLewis: no 2 are restricted [14:14:58] the disqus ones are restricted only because without config they cause issues [14:15:32] and the fatal is SoftRedirector because it's extension.json still says Disambiguator on 1.33 [14:15:46] mkay [14:16:07] * RhinosF1 now needs to decide what to do about stop forum spam [14:20:36] paladox: as I see it there's no clear issue with StopForumSpam breaking anything we just don't know whether it's stopping things properly so i'm leaning towards letting it wild [14:21:14] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.95, 7.99, 7.90 [14:21:31] ok [14:21:50] paladox: i'll enable it [14:21:56] ok [14:24:19] [02mw-config] 07RhinosF1 opened pull request 03#2742: enable stopforumspam in prod - 13https://git.io/fjAto [14:24:29] paladox: ^ [14:28:53] ok [14:29:22] [02mw-config] 07paladox closed pull request 03#2742: enable stopforumspam in prod - 13https://git.io/fjAto [14:29:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAtM [14:29:24] Done [14:29:26] [02miraheze/mw-config] 07RhinosF1 0353f8858 - enable stopforumspam in prod (#2742) a weak +1 from me [14:30:50] paladox: thx [14:31:16] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.40, 7.27, 7.37 [14:32:30] I have to revert it for now, another opper wants to take another look to be sure [14:32:39] paladox: okay [14:32:46] [02mw-config] 07paladox created branch 03revert-2742-patch-7 - 13https://git.io/vbvb3 [14:32:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-2742-patch-7 [+0/-0/±1] 13https://git.io/fjAt5 [14:32:49] [02miraheze/mw-config] 07paladox 03adb2a73 - Revert "enable stopforumspam in prod (#2742)" This reverts commit 53f8858b110d73afcdd62771df095ed09ca25693. [14:32:51] [02mw-config] 07paladox opened pull request 03#2743: Revert "enable stopforumspam in prod" - 13https://git.io/fjAtd [14:32:56] [02mw-config] 07paladox closed pull request 03#2743: Revert "enable stopforumspam in prod" - 13https://git.io/fjAtd [14:32:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAtF [14:32:59] [02miraheze/mw-config] 07paladox 03136b981 - Revert "enable stopforumspam in prod (#2742)" (#2743) This reverts commit 53f8858b110d73afcdd62771df095ed09ca25693. [14:33:14] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.95, 6.78, 7.18 [14:33:21] [02miraheze/mw-config] 07paladox deleted branch 03revert-2742-patch-7 [14:33:23] [02mw-config] 07paladox deleted branch 03revert-2742-patch-7 - 13https://git.io/vbvb3 [14:33:51] miraheze/mw-config/revert-2742-patch-7/adb2a73 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/576464241 [14:34:25] paladox: ^ looks like travis in a mood [14:34:45] and who? do you want to PM info? or use the private chat [14:35:11] heh and nope. [14:35:13] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.71, 5.16, 6.53 [14:35:24] paladox: okay [14:36:24] * RhinosF1 says whoever it is is welcome to make themselves known [14:46:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAqJ [14:46:39] [02miraheze/mw-config] 07paladox 0338cf80b - Revert "Revert "enable stopforumspam in prod (#2742)" (#2743)" This reverts commit 136b981f66ed38cd046373605d400ba9bb718e86. [14:47:15] paladox: everyone happy now? [14:47:18] yup [14:47:57] RhinosF1: there used to be a feature that allowed you to send PII to external sites for reporting purposes [14:50:04] Little did I know that this was removed in 2017, though I did use that feature years ago and enabling that feature would be a very bad idea. I needed to make sure that I didn't approve a rather dangerous extension [14:50:56] SPF|Cloud: okay, yeah that makes sense [14:54:22] (and this proved that extensions reviews are very serious.) [14:54:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAqY [14:54:51] [02miraheze/puppet] 07paladox 03faa5824 - Update storage_firewall.yaml [14:55:48] SPF|Cloud: yep [15:04:01] * RhinosF1 goes on a break for half an hour [15:05:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAqB [15:05:43] [02miraheze/puppet] 07paladox 030e822fc - Update services.pp [15:15:34] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:40:14] paladox: that u ^ [15:40:19] nope [15:42:59] paladox: can u look [15:43:04] yes [15:43:08] it is paladox [15:43:28] JohnLewis ? [15:43:50] paladox: duplicate definition on misc4 now [15:44:04] oh [15:45:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAmf [15:45:14] [02miraheze/puppet] 07paladox 0398e134f - services: Fix duplicated resource [15:46:18] fixed RhinosF1 JohnLewis [15:47:34] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:49:39] Thx :) [16:03:14] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjAm8 [16:03:15] [02miraheze/mediawiki] 07Reception123 034ab67ee - Update CreateWiki [16:09:40] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki on mw* [16:09:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:14:17] RECOVERY - lizardfs1 Disk Space on lizardfs1 is OK: DISK OK - free space: / 16922 MB (11% inode=98%); [18:28:49] JohnLewis, paladox: anyone to check an exception? [18:29:26] yes, i'm around. [18:30:07] paladox: CreateWiki, 2 mins ago [18:32:22] * paladox sees nothing related to CW 2 mins ago in the exception log [18:33:29] paladox: look around 19:26 maybe [18:33:52] Hopefully just a DB exisits From my browser playing up [18:34:29] what was the exception id it gave you? [18:34:50] paladox: create wiki doesn't give ids [18:35:17] If it through an exception, it would give you an id. [18:35:21] The exception handling is stuck on only ever showing a DB exisits error [18:35:37] paladox: Nah, JohnLewis broke it trying to improve stuff [18:36:43] It said the db exists already RhinosF1 ? [18:37:34] paladox: it says that whatever the error is as CW is broke but yes [18:38:07] Than it wont be in the logs [18:38:19] anyways, i have to go for dinner [18:38:26] so JohnLewis will have to handle this [18:38:31] paladox: ok [18:38:36] * RhinosF1 prods JohnLewis [18:38:52] if it's an exception it should still log [18:39:40] RECOVERY - lizardfs2 Disk Space on lizardfs2 is OK: DISK OK - free space: / 16917 MB (11% inode=98%); [18:42:53] JohnLewis: I'm hoping this time is an actual dB exisits error because I had to reload the page as my browser decided to cut off for some reason and stop loading [18:42:54] jobqueue [18:43:05] JohnLewis: oh that bloody thing again [18:43:21] And pls fix the exception handling for create wiki [18:44:04] only way I can is to just not make it fatal completely and return a 100% unhelpful and unambigious error message [18:44:46] but guess I'll look in a bit [18:45:21] JohnLewis: it gives a 100% unhelpful error now [18:45:33] not at all [18:45:42] like a clock which has stopped working, it's accurate. Just not always [18:46:14] JohnLewis: true, but like I've never seen it accurate [18:46:28] It could at least only display that when that is the issue [18:46:38] press create again, it'll then be accurate [18:47:03] True [18:48:37] JohnLewis: and please tell me I can log that on a quotes website [18:48:47] sure [18:48:57] Will do when I'm on laptop [19:23:03] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [19:23:49] PROBLEM - lizardfs1 Current Load on lizardfs1 is WARNING: WARNING - load average: 2.24, 3.53, 2.52 [19:25:03] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:25:48] RECOVERY - lizardfs1 Current Load on lizardfs1 is OK: OK - load average: 0.39, 2.39, 2.22 [19:49:17] i'm back [19:49:38] Hello back [19:49:58] hi [19:50:20] I have a task for you somewhere [19:56:42] ooh "somewhere" [19:57:38] JohnLewis: I lost the task [19:57:41] https://phabricator.miraheze.org/T4662 [19:57:42] [ ⚓ T4662 ManageWiki provides invalid options for DefaultTheme ] - phabricator.miraheze.org [19:58:00] As the default ones won't use the wmgUseSkin variables [19:59:17] enjoy paladox :P [19:59:33] He just pm'd me that JohnLewis :P [20:02:11] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [20:02:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjA3C [20:02:14] [02miraheze/mw-config] 07paladox 039809ea2 - theme: Only add wgDefaultTheme for vector and monobook. [20:02:16] [02mw-config] 07paladox opened pull request 03#2744: theme: Only add wgDefaultTheme for vector and monobook. - 13https://git.io/fjA3W [20:03:02] [02mw-config] 07paladox synchronize pull request 03#2744: theme: Only add wgDefaultTheme for vector and monobook. - 13https://git.io/fjA3W [20:03:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjA38 [20:03:05] [02miraheze/mw-config] 07paladox 039fbb5cc - Update LocalExtensions.php [20:03:27] miraheze/mw-config/paladox-patch-4/9809ea2 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/576541313 [20:04:11] miraheze/mw-config/paladox-patch-4/9fbb5cc - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/576541518 [20:07:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjA30 [20:07:43] [02miraheze/mw-config] 07paladox 03b17f57a - Update ManageWikiSettings.php [20:07:44] [02mw-config] 07paladox synchronize pull request 03#2744: theme: Only add wgDefaultTheme for vector and monobook. - 13https://git.io/fjA3W [20:08:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjA3E [20:08:24] [02miraheze/mw-config] 07paladox 0349c2548 - Update ManageWikiSettings.php [20:08:26] [02mw-config] 07paladox synchronize pull request 03#2744: theme: Only add wgDefaultTheme for vector and monobook. - 13https://git.io/fjA3W [20:10:00] [02mw-config] 07RhinosF1 reviewed pull request 03#2744 commit - 13https://git.io/fjA3g [20:10:13] paladox: spelling [20:10:37] oh [20:10:39] thanks, doing [20:10:48] [02mw-config] 07paladox synchronize pull request 03#2744: theme: Only add wgDefaultTheme for vector and monobook. - 13https://git.io/fjA3W [20:10:48] done [20:10:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/fjA3a [20:10:51] [02miraheze/mw-config] 07paladox 0399aa499 - Update ManageWikiSettings.php [20:11:47] Perfect [20:11:51] [02mw-config] 07paladox reviewed pull request 03#2744 commit - 13https://git.io/fjA3V [20:12:33] [02mw-config] 07paladox closed pull request 03#2744: theme: Only add wgDefaultTheme for vector and monobook. - 13https://git.io/fjA3W [20:12:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjA3w [20:12:36] [02miraheze/mw-config] 07paladox 03148f208 - theme: Only add wgDefaultTheme for vector and monobook. (#2744) * theme: Only add wgDefaultTheme for vector and monobook. * Update LocalExtensions.php * Update ManageWikiSettings.php * Update ManageWikiSettings.php * Update ManageWikiSettings.php [20:12:38] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [20:12:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [20:15:38] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.03, 6.12, 4.94 [20:15:42] great, now Gamepress needs to add the blue, green, and orange themes as options :P [20:16:41] paladox: ^ [20:17:03] I think you have to manually edit themes extension.json file [20:17:08] which we carn't do as we use git [20:17:20] based on https://www.mediawiki.org/wiki/Extension:Theme#Adding_themes [20:17:20] [ Extension:Theme - MediaWiki ] - www.mediawiki.org [20:17:33] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.48, 5.90, 4.97 [20:19:49] Gamepress already provides the themes, we just need to make them available as options [20:20:27] Oh? [20:20:46] yeah [20:21:07] ah [20:21:09] i see [20:22:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjA3y [20:22:56] [02miraheze/mw-config] 07paladox 0379a0ecc - Gamepress: Add blue, green and orange to wgDefaultTheme as themes [20:22:56] Voidwalker ^ [20:23:52] ty [20:52:47] [02mediawiki] 07RhinosF1 opened pull request 03#123: update SoftRedirector - 13https://git.io/fjAsU [20:53:04] paladox: ^ [20:53:51] Why is there a 1.34 branch? [20:54:01] Also we carn't use that until 1.34, than? [20:54:06] paladox: mjl says that will work with 1.33 [20:54:13] ok [20:55:23] [02mediawiki] 07paladox closed pull request 03#123: update SoftRedirector - 13https://git.io/fjAsU [20:55:24] [02miraheze/mediawiki] 07paladox pushed 039 commits to 03REL1_33 [+5/-1/±9] 13https://git.io/fjAsI [20:55:26] [02miraheze/mediawiki] 07RhinosF1 031ae413e - Merge pull request #1 from miraheze/REL1_33 Merge pull request #121 from RhinosF1/REL1_33 [20:55:27] [02miraheze/mediawiki] 07RhinosF1 03f7eada6 - adding StopForumSpam (T4554) [20:55:29] [02miraheze/mediawiki] 07RhinosF1 03d1b2026 - adding PageDisqus T4235 [20:55:30] [02miraheze/mediawiki] ... and 6 more commits. [20:59:36] paladox: squash [21:02:28] [02mw-config] 07RhinosF1 opened pull request 03#2745: revert #2739 - 13https://git.io/fjAsZ [21:03:54] [02mw-config] 07RhinosF1 synchronize pull request 03#2745: revert #2739 - 13https://git.io/fjAsZ [21:05:41] paladox: ^ [21:06:08] [02mw-config] 07paladox closed pull request 03#2745: revert #2739 - 13https://git.io/fjAsZ [21:06:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjAsc [21:06:11] [02miraheze/mw-config] 07RhinosF1 03927e129 - revert #2739 (#2745) * enable on test1 * unhash in mw [21:09:28] paladox: has puppet ran? and don't forget rebuildLC [21:09:38] nope it has not [21:09:54] paladox: run it pls on test1 [21:10:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAsR [21:10:33] [02miraheze/mw-config] 07paladox 03dd89f39 - Update extension-list [21:10:43] oops [21:11:06] [bb7e16ca72425a15ddddc45a] [no req] Exception from line 269 of /srv/mediawiki/w/includes/registration/ExtensionRegistry.php: /srv/mediawiki/w/extensions/SoftRedirector/extension.json is not a valid JSON file. [21:11:56] paladox: looking [21:12:21] Fixed with https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532273/ [21:12:22] [ Gerrit Code Review ] - gerrit.wikimedia.org [21:12:52] paladox: i'll poke matt [21:12:57] ok [21:13:18] paladox: he's seen [21:13:27] seen? [21:13:42] paladox: the pr on gerrit [21:13:46] ok [21:14:15] RhinosF1 https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532274/ [21:14:16] [ Gerrit Code Review ] - gerrit.wikimedia.org [21:16:19] paladox: pushing now [21:23:15] paladox: did u do that on REL1_34 [21:23:27] i did https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532274/ [21:23:27] [ Gerrit Code Review ] - gerrit.wikimedia.org [21:23:31] but hasen't been merged [21:23:38] paladox: git says everything up to date [21:24:18] Needs someone to merge https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SoftRedirector/+/532274/ [21:24:18] [ Gerrit Code Review ] - gerrit.wikimedia.org [21:24:48] paladox: k [21:25:19] paladox: on it [21:25:25] ok [21:33:33] [02mediawiki] 07RhinosF1 opened pull request 03#124: change branch - 13https://git.io/fjAs5 [21:33:43] paladox: ^ [21:33:51] don't ask about the branch [21:33:59] he says use that now [21:34:17] [02mediawiki] 07paladox closed pull request 03#124: change branch - 13https://git.io/fjAs5 [21:34:19] [02miraheze/mediawiki] 07paladox pushed 033 commits to 03REL1_33 [+1/-1/±4] 13https://git.io/fjAsd [21:34:20] [02miraheze/mediawiki] 07RhinosF1 03dab3f5b - change branch for softredir [21:34:22] [02miraheze/mediawiki] 07RhinosF1 03931be38 - rel1_35 for softredir [21:34:23] [02miraheze/mediawiki] 07paladox 039d661ca - Merge pull request #124 from RhinosF1/REL1_33 change branch [21:34:43] those branches will need to be deleted, otherwise when wikimedia branches mw core, that extension will be missed [21:34:48] as it would have the branch already [21:35:17] paladox: ? [21:35:53] when wikimedia goes to branch mw core and extensions/skins, that extension will be missed as it has REL1_34 and REL1_35 branches already. [21:36:02] paladox: I think the point is there doing it in parallel [21:36:09] therefore no need for branching [21:36:44] !log running lc on mw* [21:36:46] JohnLewis ok [21:36:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:37:21] paladox: http 500 on test1 [21:38:38] i hadn't ran git pull yet on there :) [21:39:25] paladox: k [21:39:31] it works now [21:41:02] paladox: testing now [21:41:07] ok [21:43:25] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.05, 1.30, 0.65 [21:43:44] paladox: should be okay, will bring live for everyone soon [21:43:49] ok [21:45:25] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.93, 1.17, 0.68 [21:47:00] [02mw-config] 07RhinosF1 opened pull request 03#2746: move SoftRedirector to prod (was test1) - 13https://git.io/fjAGq [21:48:44] [02mw-config] 07RhinosF1 synchronize pull request 03#2746: move SoftRedirector to prod (was test1) - 13https://git.io/fjAGq [21:49:23] paladox: ^ [21:50:26] https://phabricator.miraheze.org/T4617 hmm, there's been no security review by none of the staff [21:50:27] [ ⚓ T4617 Install Extension:SoftRedirector on WikiLaw ] - phabricator.miraheze.org [21:50:43] paladox: John [21:51:10] he moved it to reviewed and and confirmed to me I was fine to go ahead [21:53:02] JohnLewis: ^ [21:53:10] https://phabricator.miraheze.org/T4617#88910 [21:53:12] [ ⚓ T4617 Install Extension:SoftRedirector on WikiLaw ] - phabricator.miraheze.org [21:53:25] [02mw-config] 07paladox closed pull request 03#2746: move SoftRedirector to prod (was test1) - 13https://git.io/fjAGq [21:53:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjAGO [21:53:28] [02miraheze/mw-config] 07RhinosF1 0350a93b5 - move SoftRedirector to prod (was test1) (#2746) * move SoftRedirector to prod (was test1) * remove managewiki-restritcted requirement [22:03:31] paladox: is blog page enabled on test1 [22:04:12] yes [22:04:54] you can see this at https://test1.miraheze.org/wiki/Special:ManageWiki/extensions [22:04:56] [ Manage a wiki - Test1 ] - test1.miraheze.org [22:16:18] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAGz [22:16:20] [02miraheze/ssl] 07paladox 0395ab6da - Re enable events for wiki.s23.org [22:16:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjAGg [22:16:35] [02miraheze/ssl] 07paladox 0319cc0cb - Update wiki.s23.org.crt [22:17:03] paladox: any chance u can clear any of the ssl backlog? [22:18:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAGw [22:18:25] [02miraheze/ssl] 07paladox 035e66ea9 - Remove wiki.aspirantsadda.com ssl certificate Domain is no longer pointing at our nameservers or mw-lib.miraheze.org [22:18:27] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [22:18:28] [02ssl] 07paladox opened pull request 03#213: Remove wiki.aspirantsadda.com ssl certificate - 13https://git.io/fjAGr [22:18:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fjAGo [22:18:49] [02miraheze/ssl] 07paladox 0365e044b - Delete wiki.aspirantsadda.com.crt [22:18:51] [02ssl] 07paladox synchronize pull request 03#213: Remove wiki.aspirantsadda.com ssl certificate - 13https://git.io/fjAGr [22:19:16] [02ssl] 07paladox closed pull request 03#213: Remove wiki.aspirantsadda.com ssl certificate - 13https://git.io/fjAGr [22:19:18] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fjAG6 [22:19:20] [02miraheze/ssl] 07paladox 03a9037d2 - Remove wiki.aspirantsadda.com ssl certificate (#213) * Remove wiki.aspirantsadda.com ssl certificate Domain is no longer pointing at our nameservers or mw-lib.miraheze.org * Delete wiki.aspirantsadda.com.crt [22:20:27] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [22:20:28] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [22:22:46] RhinosF1 i can take a look [22:23:36] paladox: enjoy and good night - time for me to sleep [22:24:04] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [22:24:05] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjAG1 [22:24:07] [02miraheze/ssl] 07paladox 03dca9b9d - Add warmage.necrocity.co.uk ssl certificate [22:24:08] [02ssl] 07paladox opened pull request 03#214: Add warmage.necrocity.co.uk ssl certificate - 13https://git.io/fjAGM [22:24:43] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjAGD [22:24:45] [02miraheze/ssl] 07paladox 03db51f06 - Create warmage.necrocity.co.uk.crt [22:24:45] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:24:46] [02ssl] 07paladox synchronize pull request 03#214: Add warmage.necrocity.co.uk ssl certificate - 13https://git.io/fjAGM [22:25:09] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:25:13] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjAGy [22:25:15] [02miraheze/ssl] 07paladox 03ad42c1f - Add warmage.necrocity.co.uk ssl certificate (#214) * Add warmage.necrocity.co.uk ssl certificate * Create warmage.necrocity.co.uk.crt [22:25:17] [02ssl] 07paladox closed pull request 03#214: Add warmage.necrocity.co.uk ssl certificate - 13https://git.io/fjAGM [22:25:18] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [22:25:20] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [22:25:25] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:25:27] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:25:56] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:26:04] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:26:25] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.aspirantsadda.com],File[wiki.aspirantsadda.com_private] [22:29:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:29:28] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Sat 23 Nov 2019 09:15:31 PM GMT +0000. [22:29:47] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:33:04] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:33:11] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:33:30] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:33:47] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:34:01] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures