[00:24:44] Note: constant session loss problems [00:39:30] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [00:41:11] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [00:42:06] PROBLEM - mw2 Current Load on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [00:42:11] PROBLEM - mw2 php-fpm on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [00:42:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.44, 7.97, 4.87 [00:43:17] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [00:44:52] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.40, 5.37, 4.52 [00:45:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/Jv8TV [00:45:01] [02miraheze/puppet] 07paladox 0307ef3ac - varnish: Remove mw[4567] They cause the site to be slow due to the long distance to db4. [00:45:03] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS [00:45:04] [02puppet] 07paladox opened pull request 03#1243: varnish: Remove mw[4567] - 13https://git.io/Jv8Tw [00:45:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8Tr [00:45:42] [02miraheze/puppet] 07paladox 03123f389 - varnish: Remove mw[4567] (#1243) They cause the site to be slow due to the long distance to db4. [00:45:43] [02puppet] 07paladox closed pull request 03#1243: varnish: Remove mw[4567] - 13https://git.io/Jv8Tw [00:45:45] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://git.io/vbiAS [00:45:46] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [00:46:15] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [00:46:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8Ti [00:47:00] [02miraheze/puppet] 07paladox 03a3e705b - Update default.vcl [00:49:33] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [00:50:46] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.27, 5.43, 4.13 [00:50:54] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [00:50:57] RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 13 processes with command name 'php-fpm7.3' [00:51:00] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 6 backends are healthy [00:51:05] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 70154 MB (86% inode=93%); [00:51:32] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:52:32] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:52:44] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.70, 4.45, 4.03 [00:55:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/Jv8Td [00:55:14] [02miraheze/puppet] 07paladox 03d4034f5 - mw[123] increase fpm_workers_multiplier to 2.0 [00:55:15] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS [00:55:17] [02puppet] 07paladox opened pull request 03#1244: mw[123] increase fpm_workers_multiplier to 2.0 - 13https://git.io/Jv8TF [00:55:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/Jv8Tb [00:55:31] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures [00:55:32] [02miraheze/puppet] 07paladox 035e99b5f - Update mw2.yaml [00:55:34] [02puppet] 07paladox synchronize pull request 03#1244: mw[123] increase fpm_workers_multiplier to 2.0 - 13https://git.io/Jv8TF [00:55:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/Jv8TN [00:55:44] [02miraheze/puppet] 07paladox 038687579 - Update mw3.yaml [00:55:46] [02puppet] 07paladox synchronize pull request 03#1244: mw[123] increase fpm_workers_multiplier to 2.0 - 13https://git.io/Jv8TF [00:55:53] [02puppet] 07paladox closed pull request 03#1244: mw[123] increase fpm_workers_multiplier to 2.0 - 13https://git.io/Jv8TF [00:55:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jv8TA [00:55:56] [02miraheze/puppet] 07paladox 03ecc9c6e - mw[123] increase fpm_workers_multiplier to 2.0 (#1244) * mw[123] increase fpm_workers_multiplier to 2.0 * Update mw2.yaml * Update mw3.yaml [00:55:57] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://git.io/vbiAS [00:55:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [00:56:10] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:56:52] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:57:32] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:00:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8Tp [01:00:58] [02miraheze/puppet] 07paladox 0325cc1f8 - matomo: Update to 3.13.2 [01:52:36] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [01:54:44] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.81, 4.51, 2.67 [01:58:45] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet last ran 1 hour ago [02:03:09] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.58, 1.39, 1.84 [02:05:14] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.32, 1.04, 1.65 [02:23:10] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.60, 7.24, 6.58 [02:27:05] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.18, 6.67, 6.52 [02:52:47] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.27, 7.37, 6.63 [02:54:45] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.26, 6.00, 6.27 [03:04:27] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet last ran 1 hour ago [03:52:10] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:08:57] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: Full, 4431 files, 9.358MB, 2020-01-31 06:08:00 (2.3 weeks ago) [06:14:36] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:26:38] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3387 MB (14% inode=94%); [07:13:03] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:44:11] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet last ran 1 hour ago [09:12:08] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.52, 7.41, 6.48 [09:12:52] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.06, 7.19, 6.05 [09:14:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.25, 5.98, 5.84 [09:16:04] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.81, 5.94, 6.30 [09:32:10] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [11:45:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8nr [11:45:09] [02miraheze/services] 07MirahezeSSLBot 03a83ea7d - BOT: Updating services config for wikis [11:52:17] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.94, 6.87, 4.79 [11:54:15] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.72, 4.86, 4.41 [12:55:47] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet last ran 12 hours ago [13:08:51] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.78, 6.77, 5.46 [13:10:53] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.63, 6.08, 5.49 [13:11:48] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.74, 9.97, 7.79 [13:14:30] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:15:43] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.70, 7.83, 7.74 [13:19:39] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.37, 4.68, 6.23 [13:19:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-1244-paladox-patch-10 [+0/-0/±3] 13https://git.io/Jv8Cu [13:19:54] [02miraheze/puppet] 07paladox 03ee3fc3e - Revert "mw[123] increase fpm_workers_multiplier to 2.0 (#1244)" This reverts commit ecc9c6e14e553ebcd1225f293e52884bdabd4629. [13:19:56] [02puppet] 07paladox created branch 03revert-1244-paladox-patch-10 - 13https://git.io/vbiAS [13:19:57] [02puppet] 07paladox opened pull request 03#1245: Revert "mw[123] increase fpm_workers_multiplier to 2.0" - 13https://git.io/Jv8Cz [13:20:02] [02puppet] 07paladox closed pull request 03#1245: Revert "mw[123] increase fpm_workers_multiplier to 2.0" - 13https://git.io/Jv8Cz [13:20:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jv8Cg [13:20:05] [02miraheze/puppet] 07paladox 0376ca8a1 - Revert "mw[123] increase fpm_workers_multiplier to 2.0 (#1244)" (#1245) This reverts commit ecc9c6e14e553ebcd1225f293e52884bdabd4629. [13:20:07] [02puppet] 07paladox deleted branch 03revert-1244-paladox-patch-10 - 13https://git.io/vbiAS [13:20:08] [02miraheze/puppet] 07paladox deleted branch 03revert-1244-paladox-patch-10 [13:34:31] PROBLEM - mw2 Current Load on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [13:34:35] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [13:35:23] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [13:36:09] PROBLEM - mw2 php-fpm on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [13:42:09] RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 13 processes with command name 'php-fpm7.3' [13:42:36] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures [13:43:16] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 69981 MB (86% inode=93%); [13:44:21] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.77, 4.38, 4.64 [14:30:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jv8WA [14:30:32] [02miraheze/mw-config] 07paladox 03fe911a6 - Database: Make connection persistent [14:30:34] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [14:30:51] [02mw-config] 07paladox opened pull request 03#2885: Database: Make connection persistent - 13https://git.io/Jv8Wx [14:30:52] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.17, 6.82, 5.90 [14:31:46] !log restart php on mw[123] [14:31:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:32:49] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.04, 5.79, 5.71 [14:38:23] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.026 second response time [14:38:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [14:38:54] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:39:25] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw1 mw2 mw3 lizardfs6 [14:39:37] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [14:39:53] PROBLEM - db4 MySQL on db4 is CRITICAL: Too many connections [14:40:01] hmm [14:40:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [14:40:21] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:40:40] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.046 second response time [14:41:46] !log reboot lizardfs6 [14:41:49] RECOVERY - db4 MySQL on db4 is OK: Uptime: 78644 Threads: 80 Questions: 81988034 Slow queries: 2946901 Opens: 13088636 Flush tables: 1 Open tables: 1000 Queries per second avg: 1042.521 [14:42:38] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:42:46] !log [14:41:46] <+paladox> !log reboot lizardfs6 [14:42:50] PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:43:09] !log hacked lizardfs6 Database.php (added persitents mode and removed) [14:43:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:43:44] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:01] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 12% [14:44:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:44:19] !log [14:41:46] <+paladox> !log reboot lizardfs6 [14:44:25] !log [14:43:09] <+paladox> !log hacked lizardfs6 Database.php (added persitents mode and removed) [14:44:26] PROBLEM - lizardfs6 JobChron Service on lizardfs6 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [14:44:50] PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Full, 2336892 files, 199.8GB, 2020-01-30 17:58:00 (2.4 weeks ago) [14:44:52] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 6 backends are healthy [14:45:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:45:42] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 20532 bytes in 0.708 second response time [14:45:56] PROBLEM - lizardfs6 JobRunner Service on lizardfs6 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [14:47:59] PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:48:06] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 20533 bytes in 3.389 second response time [14:48:11] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 20533 bytes in 1.049 second response time [14:49:59] PROBLEM - bacula1 Bacula Static on bacula1 is WARNING: WARNING: Full, 2336892 files, 199.8GB, 2020-01-30 17:58:00 (2.4 weeks ago) [14:50:09] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [14:50:32] RECOVERY - lizardfs6 JobRunner Service on lizardfs6 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [14:51:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 2607:5300:205:200::17f6/cpweb [14:51:07] RECOVERY - lizardfs6 JobChron Service on lizardfs6 is OK: PROCS OK: 1 process with args 'redisJobChronService' [14:52:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [14:52:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:59:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv8l6 [14:59:43] [02miraheze/mw-config] 07paladox 0343a611a - Database: Set DBO_COMPRESS flag [14:59:45] [02mw-config] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbvb3 [14:59:46] [02mw-config] 07paladox opened pull request 03#2886: Database: Set DBO_COMPRESS flag - 13https://git.io/Jv8lP [15:01:06] miraheze/mw-config/paladox-patch-5/43a611a - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/651124384 [15:01:36] [02mw-config] 07paladox closed pull request 03#2886: Database: Set DBO_COMPRESS flag - 13https://git.io/Jv8lP [15:01:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8lH [15:01:39] [02miraheze/mw-config] 07paladox 033959f7d - Database: Set DBO_COMPRESS flag (#2886) [15:01:40] [02mw-config] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbvb3 [15:01:42] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-5 [15:08:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv88e [15:08:33] [02miraheze/mw-config] 07paladox 03bad82fa - Database: Set DBO_DEFAULT flag [15:08:35] [02mw-config] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbvb3 [15:08:36] [02mw-config] 07paladox opened pull request 03#2887: Database: Set DBO_DEFAULT flag - 13https://git.io/Jv88v [16:12:52] JohnLewis: any thoughts on https://www.mediawiki.org/wiki/Topic:Veb46gqr1kbvn83y? Shall i create a phab task? [16:12:53] [ Facebook pages not found on Extension talk:RottenLinks ] - www.mediawiki.org [16:22:57] RhinosF1; the extensions checks HTTP headers, if Facebook returns 404, there’s not a lot that can be done [16:23:11] JohnLewis: Thanks [16:23:51] Perhaps maybe a blacklist I guess though? [16:25:41] JohnLewis: we could blacklist it as an option. Have the blacklist behind a feature flag and then allow users to opt in [16:26:10] Yeah that was my idea [16:26:31] Shall I phab that? [16:28:04] Can do [16:28:53] Consider it done [17:21:45] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [17:23:46] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2652 MB (11% inode=94%); [17:27:45] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [18:08:44] [02puppet] 07paladox synchronize pull request 03#1235: mariadb: Bind to 0.0.0.0 - 13https://git.io/Jvlik [18:08:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv8Ek [18:08:47] [02miraheze/puppet] 07paladox 0307d3962 - Update mw.cnf.erb [18:08:51] [02puppet] 07paladox edited pull request 03#1235: mariadb: Bind to :: - 13https://git.io/Jvlik [18:10:18] [02puppet] 07paladox closed pull request 03#1235: mariadb: Bind to :: - 13https://git.io/Jvlik [18:10:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8Eq [18:10:21] [02miraheze/puppet] 07paladox 03a568548 - mariadb: Bind to :: (#1235) * mariadb: Bind to 0.0.0.0 This will allow port 3306 to be served over both ipv6 and ipv4. * Update mw.cnf.erb * Update mw.cnf.erb * Update mw.cnf.erb [18:10:23] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [18:10:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:11:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8Es [18:11:53] [02miraheze/puppet] 07paladox 03df96152 - Update mw4.yaml [18:12:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8EC [18:12:37] [02miraheze/mw-config] 07paladox 03402a1f7 - Update Database.php [18:14:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8EE [18:14:49] [02miraheze/mw-config] 07paladox 030b51a97 - Update Database.php [19:03:01] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.03, 2.13, 1.41 [19:03:37] JohnLewis, PuppyKun, Voidwalker, Reception123, Zppix: https://meta.miraheze.org/w/index.php?diff=96588&oldid=96578&rcid=378578 [19:03:39] [ Difference between revisions of "Stewards' noticeboard" - Miraheze Meta ] - meta.miraheze.org [19:04:15] oh, I think I recall that wiki [19:05:00] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.84, 1.70, 1.34 [19:05:02] Voidwalker: can you handle that issue? [19:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8zk [19:05:09] [02miraheze/services] 07MirahezeSSLBot 030adf7c9 - BOT: Updating services config for wikis [19:05:16] as a non-steward, since Void is around I'll leave it to him :) [19:05:24] Voidwalker: he’s messing managewiki around [19:05:37] (change visibility) 18:58, 16 February 2020 Landscape talk contribs block changed group membership for Landscape from confirmed user to confirmed user, administrator and bureaucrat [19:05:37] (change visibility) 18:56, 16 February 2020 No Face talk contribs block changed group membership for Landscape from (none) to confirmed user [19:05:43] RhinosF1: but there could be a technical aspect to it [19:05:53] I'll check what happened with MW [19:06:01] Reception123: i thought worse case you might have been able to take staff action to fix and lock managewiki until stew turned up [19:06:08] RhinosF1: https://amazingyoutubers.miraheze.org/wiki/Special:ManageWiki/permissions/confirmed#mw-section-group [19:06:09] [ Permission error - HACKED ] - amazingyoutubers.miraheze.org [19:06:19] RhinosF1: yes, would've done that if Voidwalker wasn't around as an emergency measure [19:06:25] Reception123: permission error [19:06:37] Can you make it public back? [19:06:43] RhinosF1: yeah, just realized but ManageWiki/permissions shows confirmed users as being able to add admin/crat [19:06:46] RhinosF1: I'll let Voidwalker do that [19:07:02] Reception123: ouch [19:07:08] 19:58, 4 December 2019 No Face talk contribs block changed group metadata for Special:ManageWiki/permissions/confirmed: add rights (none); removed rights (none); added addgroups sysop, bureaucrat; removed addgroups (none); added removegroups sysop, bureaucrat; removed removegroups (none); added addgroupstoself (none); removed addgroupstoself (none); added removegroupsfromself (none); removed removegroupsfromself (none); modified [19:07:08] autopromotion (no) (.) [19:07:20] so it's the user's fault [19:07:25] Hmm [19:19:47] Reception123: JohnLewis Voidwalker see Recent Changes [19:20:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8zW [19:20:10] [02miraheze/services] 07MirahezeSSLBot 03ae01ab5 - BOT: Updating services config for wikis [19:23:24] PROBLEM - interno.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'interno.guiasdobrasil.com.br' expired on Sat 16 Nov 2019 01:39:54 PM GMT +0000. [19:28:58] !log removed spam from cw_requests (inappropriate comment) [19:29:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:42:12] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.28, 2.63, 1.80 [19:46:11] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.28, 1.90, 1.67 [19:48:11] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.78, 1.53, 1.57 [20:31:31] !log re-inserted cw_requests from db6 after misclick [20:31:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:46:45] paladox: Error deleting file: Rayman small 3678.jpg [20:46:45] Error deleting file: An unknown error occurred in storage backend "local-backend". [20:46:55] From SN thread [20:48:16] ok [20:49:17] paladox: not ok. It’s 2 days on now. Fix it. [20:57:56] RhinosF1 2 days?! That user only reported that only like ~30 mins ago [21:02:23] paladox: they only gave the error. I’m going to say now it came from the migration [21:13:25] To be fair, action can only reasonably be expected in a sufficient time from reporting [21:16:32] JohnLewis: when no one could upload. Could have been checked. [21:17:10] My understanding is that was an intentional thing though [21:21:17] JohnLewis: not for as long as it happened [21:25:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8aG [21:25:08] [02miraheze/services] 07MirahezeSSLBot 034e72bb4 - BOT: Updating services config for wikis [22:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8VT [22:05:09] [02miraheze/services] 07MirahezeSSLBot 038e89a9a - BOT: Updating services config for wikis [22:50:37] PROBLEM - wiesepedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiesepedia.ga' expires in 15 day(s) (Tue 03 Mar 2020 10:47:41 PM GMT +0000). [22:50:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8wY [22:50:52] [02miraheze/ssl] 07MirahezeSSLBot 03ce0633d - Bot: Update SSL cert for wiesepedia.ga [22:55:56] !log set read_only=1 on db6 [22:56:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:02:37] RECOVERY - wiesepedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiesepedia.ga' will expire on Sat 16 May 2020 09:50:45 PM GMT +0000. [23:48:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8rN [23:49:00] [02miraheze/mw-config] 07paladox 036771802 - Update LocalSettings.php [23:51:56] !log MariaDB [allthetropeswiki]> truncate objectcache; [23:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master