[00:06:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:08:03] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:08:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4v8 [00:08:04] [02miraheze/puppet] 07paladox 0348f5f16 - Update config.yaml [00:08:21] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:09:24] PROBLEM - lizardfs4 Puppet on lizardfs4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [00:09:38] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:09:39] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:10:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4v4 [00:10:31] [02miraheze/puppet] 07paladox 0343a1053 - Update lizardfs6.yaml [00:13:27] !log restart lizardfs-master [00:14:42] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:16:21] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 10.87, 9.45, 5.72 [00:19:33] PROBLEM - lizardfs4 Puppet on lizardfs4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [00:24:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4v6 [00:24:24] [02miraheze/mw-config] 07paladox 03c45987f - Update LocalSettings.php [00:31:09] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.17, 1.98, 3.27 [00:36:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [00:43:03] !log restart php-fpm on mw1 [00:43:23] !log restart php-fpm on mw2 [00:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:43:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:46:14] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:46:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:56:12] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:56:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:57:03] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:57:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4vS [00:58:00] [02miraheze/puppet] 07paladox 03ade9c5a - nginx: Increase worker_connections [01:15:21] !log reboot misc4 - OOM [01:15:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:25:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4vx [01:25:57] [02miraheze/mw-config] 07paladox 03d72b469 - Update LocalSettings.php [01:28:54] paladox: You here? [01:28:59] yup [01:29:18] u want to join my personal channel ##Examknow [01:29:58] No, thanks. [01:30:06] okay [01:40:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4fI [01:40:30] [02miraheze/puppet] 07paladox 0325dea59 - nginx: Put time at the start Makes it easier to compare times. [01:51:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4fq [01:51:22] [02miraheze/puppet] 07paladox 03c9b3f69 - Update mediawiki.conf [01:52:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4fm [01:52:04] [02miraheze/puppet] 07paladox 030a6b339 - Update mediawiki.conf [01:56:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4fO [01:56:35] [02miraheze/puppet] 07paladox 034fbb9fc - Update mediawiki.conf [05:32:30] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2608 MB (10% inode=94%); [06:25:04] RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 334656 files, 62.59GB, 2019-10-16 06:23:00 (2.1 minutes ago) [06:27:41] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3323 MB (13% inode=94%); [06:53:48] Hello XavierV! If you have any questions, feel free to ask and someone should answer soon. [07:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4IJ [07:40:11] [02miraheze/services] 07MirahezeSSLBot 034326456 - BOT: Updating services config for wikis [07:42:48] Hello Uomovariabile! If you have any questions, feel free to ask and someone should answer soon. [11:57:22] https://meta.miraheze.org/wiki/Special:Contributions/Fukucunt - SPF|Cloud PuppyKun requesting global steward lock per violation of username policy [11:57:24] [ User contributions for Fukucunt - Miraheze Meta ] - meta.miraheze.org [14:40:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4GF [14:40:29] [02miraheze/mw-config] 07paladox 03a877dde - Update LocalSettings.php [14:43:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4Gb [14:43:29] [02miraheze/mw-config] 07paladox 03fd69306 - Update LocalSettings.php [14:47:44] 5 03 [14:48:01] known, should recover [14:48:18] paladox :) [14:53:31] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [14:53:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je4Zq [14:53:34] [02miraheze/puppet] 07paladox 039e88bb2 - mediawiki: Increase write threads for lizardfs [14:53:36] [02puppet] 07paladox opened pull request 03#1104: mediawiki: Increase write threads for lizardfs - 13https://git.io/Je4Zm [14:53:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:54:30] !log depool and repool mw1 [14:54:48] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [14:55:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:55:49] !log depool and repool mw2 [14:56:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:57:04] !log depool and repool mw3 [14:57:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:58:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:59:25] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:03:46] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [15:03:59] !log running sudo chown -R www-data:www-data /mnt/mediawiki-static on mw1 [15:04:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:05:43] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [15:07:12] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [15:09:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4ZV [15:09:54] [02miraheze/puppet] 07paladox 031d9d120 - Revert "nginx: Put time at the start" This reverts commit 25dea593f56eea97183596912267d7c5b5e9fd0d. [15:13:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4ZX [15:13:46] [02miraheze/mw-config] 07paladox 03afb5d9a - Update LocalSettings.php [15:19:29] and we have 503s [15:19:51] again, after site performance took a massive hit, 503s usually follow [15:21:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [15:22:20] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:31] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:46] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [15:23:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:24:18] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [15:24:53] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [15:25:28] k6ka yeh [15:25:41] we are seeing spikes in our grafana graph for active connections [15:29:00] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.004 second response time [15:29:06] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.389 second response time [15:29:07] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:29:42] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:30:16] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:32:49] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 2.028 second response time [15:42:34] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [15:43:47] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [15:43:58] sigh [15:44:19] was in the middle of an edit [15:44:27] luckily i saved the text to pastebin [15:44:32] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [15:46:08] k6ka sorry that this is happening :( [15:49:02] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4nF [15:53:55] [02miraheze/mw-config] 07paladox 0384fc431 - Disable wgGenerateThumbnailOnParse [15:54:14] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.426 second response time [15:55:19] !log rolling restart of php-fpm on mw[123] [15:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:57:25] !log restart nginx on mw1 [15:57:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:57:39] !log restart nginx on mw2 [15:57:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:58:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:58:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:59:14] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:59:56] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:00:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:03:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4cv [16:03:15] [02miraheze/mw-config] 07paladox 0359b294c - Update LocalSettings.php [16:10:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4cq [16:10:27] [02miraheze/puppet] 07paladox 0381bcf13 - Update mediawiki.conf [16:12:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4c3 [16:12:32] [02miraheze/puppet] 07paladox 03cef70eb - Update mediawiki.conf [16:27:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4cX [16:27:50] [02miraheze/mw-config] 07paladox 030723fc3 - Fix [16:28:36] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Je4c1 [16:28:38] [02miraheze/mw-config] 07paladox 03ff8c0b8 - Revert "Fix" This reverts commit 0723fc3548e35c569481fc2fce93a7593ea5365e. [16:28:39] [02miraheze/mw-config] 07paladox 03341519b - Revert "Update LocalSettings.php" This reverts commit 59b294ca9b953a44f6c7957874c50dddb90bc818. [16:29:48] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.24, 7.97, 5.90 [16:30:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.71, 6.97, 5.87 [16:31:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4cH [16:31:27] [02miraheze/puppet] 07paladox 03e293789 - Update mediawiki.conf [16:34:23] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.63, 6.62, 5.92 [16:35:27] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.16, 6.52, 5.96 [18:23:20] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.11, 9.05, 7.52 [18:32:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.91, 7.28, 7.42 [18:41:26] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.22, 6.22, 6.77 [19:11:17] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [19:11:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4Wb [19:11:20] [02miraheze/puppet] 07paladox 03c5b6371 - nginx: Introduce nginx_worker_processes with the default set at auto [19:11:22] [02puppet] 07paladox opened pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:13:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4Wj [19:13:09] [02miraheze/puppet] 07paladox 035163ad4 - Update init.pp [19:13:11] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:14:13] [02puppet] 07paladox edited pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:14:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lf [19:14:36] [02miraheze/puppet] 07paladox 03fa01bd6 - Update mw1.yaml [19:14:38] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:14:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lJ [19:14:49] [02miraheze/puppet] 07paladox 0396f4ebe - Update mw2.yaml [19:14:50] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:14:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lU [19:14:58] [02miraheze/puppet] 07paladox 0312b832e - Update mw3.yaml [19:15:00] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:15:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lT [19:15:18] [02miraheze/puppet] 07paladox 03093206c - Update cp4.yaml [19:15:19] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:15:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lk [19:15:31] [02miraheze/puppet] 07paladox 033f7499e - Update cp3.yaml [19:15:32] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:15:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lI [19:15:43] [02miraheze/puppet] 07paladox 03d6060a5 - Update cp2.yaml [19:15:44] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:16:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je4lL [19:16:09] [02miraheze/puppet] 07paladox 034d0ac2e - Update nginx.conf.erb [19:16:11] [02puppet] 07paladox synchronize pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:16:35] [02puppet] 07paladox edited pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [19:42:41] PROBLEM - www.schulwiki.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.schulwiki.de' expires in 15 day(s) (Fri 01 Nov 2019 07:35:33 PM GMT +0000). [19:42:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lz [19:42:57] [02miraheze/ssl] 07MirahezeSSLBot 03fb87d61 - Bot: Update SSL cert for www.schulwiki.de [19:44:52] PROBLEM - wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Fri 01 Nov 2019 07:35:18 PM GMT +0000). [19:45:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lV [19:45:12] [02miraheze/ssl] 07MirahezeSSLBot 0399568d1 - Bot: Update SSL cert for wikiescola.com.br [19:46:17] PROBLEM - www.wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Fri 01 Nov 2019 07:35:18 PM GMT +0000). [19:48:40] PROBLEM - www.openonderwijs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.openonderwijs.org' expires in 15 day(s) (Fri 01 Nov 2019 07:37:56 PM GMT +0000). [19:48:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lr [19:48:55] [02miraheze/ssl] 07MirahezeSSLBot 03510c08a - Bot: Update SSL cert for www.openonderwijs.org [19:49:16] PROBLEM - www.evanswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.evanswiki.org' expires in 15 day(s) (Fri 01 Nov 2019 07:41:16 PM GMT +0000). [19:49:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lo [19:49:34] [02miraheze/ssl] 07MirahezeSSLBot 034433b7b - Bot: Update SSL cert for www.evanswiki.org [19:49:45] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Fri 01 Nov 2019 07:40:20 PM GMT +0000). [19:50:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lK [19:50:05] [02miraheze/ssl] 07MirahezeSSLBot 03406d3d1 - Bot: Update SSL cert for marinebiodiversitymatrix.org [19:51:45] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Fri 01 Nov 2019 07:40:20 PM GMT +0000). [19:53:13] RECOVERY - www.openonderwijs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.openonderwijs.org' will expire on Tue 14 Jan 2020 06:48:48 PM GMT +0000. [19:53:49] RECOVERY - www.evanswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.evanswiki.org' will expire on Tue 14 Jan 2020 06:49:27 PM GMT +0000. [19:53:59] RECOVERY - wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Tue 14 Jan 2020 06:45:00 PM GMT +0000. [19:54:17] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 14 Jan 2020 06:49:57 PM GMT +0000. [19:55:19] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Fri 01 Nov 2019 07:46:23 PM GMT +0000). [19:55:26] RECOVERY - www.wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Tue 14 Jan 2020 06:45:00 PM GMT +0000. [19:55:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4l9 [19:55:34] [02miraheze/ssl] 07MirahezeSSLBot 03eb62dae - Bot: Update SSL cert for dariawiki.org [19:56:16] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 14 Jan 2020 06:49:57 PM GMT +0000. [19:56:16] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Fri 01 Nov 2019 07:46:23 PM GMT +0000). [19:56:17] RECOVERY - www.schulwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.schulwiki.de' will expire on Tue 14 Jan 2020 06:42:50 PM GMT +0000. [19:56:19] PROBLEM - www.bushcraftpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bushcraftpedia.org' expires in 15 day(s) (Fri 01 Nov 2019 07:47:15 PM GMT +0000). [19:56:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lQ [19:56:33] [02miraheze/ssl] 07MirahezeSSLBot 032107533 - Bot: Update SSL cert for www.bushcraftpedia.org [19:57:24] PROBLEM - www.bestpractice.technology - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bestpractice.technology' expires in 15 day(s) (Fri 01 Nov 2019 07:49:01 PM GMT +0000). [19:57:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4l7 [19:57:40] [02miraheze/ssl] 07MirahezeSSLBot 03e7f2064 - Bot: Update SSL cert for www.bestpractice.technology [19:59:20] PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Fri 01 Nov 2019 07:50:54 PM GMT +0000). [20:00:30] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Fri 01 Nov 2019 07:50:54 PM GMT +0000). [20:00:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lF [20:00:47] [02miraheze/ssl] 07MirahezeSSLBot 03aef75e8 - Bot: Update SSL cert for allthetropes.org [20:02:53] PROBLEM - wintermoor.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wintermoor.net' expires in 15 day(s) (Fri 01 Nov 2019 07:52:23 PM GMT +0000). [20:03:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4lA [20:03:08] [02miraheze/ssl] 07MirahezeSSLBot 03f1bcd57 - Bot: Update SSL cert for wintermoor.net [20:04:18] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Tue 14 Jan 2020 06:55:26 PM GMT +0000. [20:04:58] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Tue 14 Jan 2020 06:55:26 PM GMT +0000. [20:05:13] RECOVERY - www.bushcraftpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bushcraftpedia.org' will expire on Tue 14 Jan 2020 06:56:26 PM GMT +0000. [20:06:22] RECOVERY - www.bestpractice.technology - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bestpractice.technology' will expire on Tue 14 Jan 2020 06:57:32 PM GMT +0000. [20:12:45] RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 14 Jan 2020 07:00:38 PM GMT +0000. [20:13:49] RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 14 Jan 2020 07:00:38 PM GMT +0000. [20:16:13] RECOVERY - wintermoor.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wintermoor.net' will expire on Tue 14 Jan 2020 07:03:00 PM GMT +0000. [20:46:25] !log re-enabled puppet on test1 [20:46:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:44:55] [02puppet] 07paladox closed pull request 03#1104: mediawiki: Increase write threads for lizardfs - 13https://git.io/Je4Zm [21:44:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je44c [21:44:58] [02miraheze/puppet] 07paladox 0358f0f34 - mediawiki: Increase write threads for lizardfs (#1104) [21:44:59] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [21:45:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [21:45:14] [02puppet] 07paladox closed pull request 03#1105: nginx: Introduce nginx_worker_processes with the default set at auto - 13https://git.io/Je4WN [21:45:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/Je44C [21:45:17] [02miraheze/puppet] 07paladox 035ff61ac - nginx: Introduce nginx_worker_processes with the default set at auto (#1105) * nginx: Introduce nginx_worker_processes with the default set at auto * Update init.pp * Update mw1.yaml * Update mw2.yaml * Update mw3.yaml * Update cp4.yaml * Update cp3.yaml * Update cp2.yaml * Update nginx.conf.erb [21:45:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:45:20] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [21:46:10] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [21:49:38] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:50:13] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [22:04:57] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je44i [22:04:59] [02miraheze/puppet] 07Southparkfan 03add6262 - Update lizardfs6 icinga ip [22:08:52] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:08:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je44M [22:08:55] [02miraheze/puppet] 07paladox 038da9ae7 - monitoring: Support custom ips in hosts [22:09:40] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:09:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je44D [22:09:43] [02miraheze/puppet] 07paladox 0320744df - monitoring: Support custom ips in hosts [22:09:45] [02puppet] 07paladox opened pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:10:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je44S [22:10:23] [02miraheze/puppet] 07paladox 03bb2dcb9 - Update lizardfs6.yaml [22:10:24] [02puppet] 07paladox synchronize pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:11:38] [02puppet] 07paladox synchronize pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:11:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je449 [22:11:42] [02miraheze/puppet] 07paladox 038012cac - Update hosts.pp [22:15:47] [02puppet] 07paladox synchronize pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:15:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je44Q [22:15:50] [02miraheze/puppet] 07paladox 034028fb3 - Update hosts.pp [22:16:02] [02puppet] 07paladox synchronize pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:16:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je447 [22:16:05] [02miraheze/puppet] 07paladox 036096f00 - Update lizardfs6.yaml [22:16:38] [02puppet] 07paladox synchronize pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:16:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je445 [22:16:41] [02miraheze/puppet] 07paladox 039724d73 - Update hosts.pp [22:16:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [22:16:57] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [22:16:58] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [22:17:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [22:17:01] [02miraheze/puppet] 07paladox deleted branch 03Reception123-patch-2 [22:17:03] [02puppet] 07paladox deleted branch 03Reception123-patch-2 - 13https://git.io/vbiAS [22:17:57] [02puppet] 07paladox closed pull request 03#1106: monitoring: Support custom ips in hosts - 13https://git.io/Je44y [22:17:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je44F [22:18:00] [02miraheze/puppet] 07paladox 038a6e958 - monitoring: Support custom ips in hosts (#1106) * monitoring: Support custom ips in hosts * Update lizardfs6.yaml * Update hosts.pp * Update hosts.pp * Update lizardfs6.yaml * Update hosts.pp [22:20:09] RECOVERY - Host lizardfs6 is UP: PING OK - Packet loss = 0%, RTA = 17.89 ms [22:20:13] PROBLEM - lizardfs6 Lizardfs Master Port 2 on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:13] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:53] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 0.21, 0.12, 0.04 [22:20:55] RECOVERY - lizardfs6 Disk Space on lizardfs6 is OK: DISK OK - free space: / 54796 MB (76% inode=92%); [22:20:58] PROBLEM - lizardfs6 Lizardfs Master Port 1 on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:58] PROBLEM - lizardfs6 lizard.miraheze.org HTTPS on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:58] PROBLEM - lizardfs6 Lizardfs Chunkserver Port on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:58] PROBLEM - lizardfs6 SSH on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:21:00] RECOVERY - lizardfs6 lizard.miraheze.org HTTPS on lizardfs6 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 381 bytes in 0.076 second response time [22:21:05] RECOVERY - lizardfs6 Lizardfs Chunkserver Port on lizardfs6 is OK: TCP OK - 0.015 second response time on 51.158.176.139 port 9422 [22:21:08] RECOVERY - lizardfs6 Lizardfs Master Port 1 on lizardfs6 is OK: TCP OK - 0.018 second response time on 51.158.176.139 port 9419 [22:21:11] RECOVERY - lizardfs6 Lizardfs Master Port 2 on lizardfs6 is OK: TCP OK - 0.016 second response time on 51.158.176.139 port 9420 [22:21:27] RECOVERY - lizardfs6 SSH on lizardfs6 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u1 (protocol 2.0) [22:22:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:22:39] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [22:23:45] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:24:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je44x [22:24:18] [02miraheze/puppet] 07paladox 0396c1d30 - lizardfs: Increase NR_OF_HDD_WORKERS_PER_NETWORK_WORKER to 8 for the chunkservers [22:27:54] !log restart lizardfs-chunkserver on lizardfs[45] [22:27:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:28:38] !log restart lizardfs-chunkserver on lizardfs6 [22:28:41] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 20 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:31:08] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:31:16] [02mw-config] 07GustaveLondon776 opened pull request 03#2776: Allow Steward the right of the block - 13https://git.io/Je4BJ [22:44:47] [02mw-config] 07JohnFLewis commented on pull request 03#2776: Allow Steward the right of the block - 13https://git.io/Je4Bs [22:44:49] [02mw-config] 07JohnFLewis closed pull request 03#2776: Allow Steward the right of the block - 13https://git.io/Je4BJ [22:49:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4Bc [22:49:08] [02miraheze/puppet] 07paladox 03b6458b1 - varnish: Remove some UA from rate limiter [23:01:03] k6ka we think we've found the problem, and are seeing improvements in performance. Please do let us know if you see frequent 503s/very slow performance. [23:01:13] (we've deployed a fix) [23:01:30] 👍 [23:02:06] we are still monitoring. But if things stay up for 2 days then it's definitly resolved :P [23:19:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.15, 6.77, 6.24 [23:21:21] paladox: ^ [23:21:27] ;) [23:21:32] mutante yeh :( [23:21:40] we run the jobrunner on there [23:21:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.17, 6.42, 6.17 [23:21:54] :) [23:22:18] they have 4 cores at 3ghz [23:22:20] paladox: just that the load avg is 7 does not mean it's bad [23:22:28] as long as the job queue is not growing [23:22:31] and it doesnt go down [23:22:32] i guess [23:22:45] yeh [23:22:56] maybe just change the parameters of the icinga check [23:23:04] to 100 *gg* [23:23:04] it's the old redis based one :P [23:23:08] mutante lol [23:23:23] what is replacing that? [23:23:57] paladox: i wonder what would happen if a miraheze user wants to backup en.wikipedia [23:24:07] mutante wikimedia replace it with kafka i think [23:24:19] i mean.. be a mirror [23:24:19] oh good lord, we'll run out of space :P [23:24:30] how much does it cost?:) [23:24:47] for the hdd option it starts at £3 [23:24:49] for 125GB [23:25:01] prices are fun [23:25:04] 160gb [23:25:06] do you have swift? [23:25:14] we tried that [23:25:18] swift is unstable :P [23:25:26] ceph? [23:25:44] that required kvm at the time when i want to try it [23:25:48] but ceph is like swift [23:25:51] and object store [23:25:58] we use lizardfs mutante [23:26:04] brings superb performance [23:26:13] oh, that replaced the object storage? [23:26:16] interesting [23:26:45] yeh, we had to find a quck option [23:27:45] https://github.com/lizardfs/lizardfs/issues/805 [23:27:46] [ Is LizardFS development still alive? · Issue #805 · lizardfs/lizardfs · GitHub ] - github.com [23:27:48] mutante comes with a UI :P [23:27:52] oh lol [23:27:56] you found that :D [23:28:01] development is back [23:28:02] heh :) [23:28:05] company was took over [23:28:11] interesting [23:28:13] mutante LizardFS is a fork of MooseFS :P [23:28:18] paladox: i just learned that [23:28:24] paladox: close the ticket as resolved :) [23:28:26] :D [23:28:32] oh, it is closed [23:28:32] hah [23:28:37] mutante i told bstorm about lizard for the cloud [23:28:45] 20 days ago they talk about a slack channel [23:28:56] still a chance to stop slack [23:29:02] and make them use IRC or at least discord [23:29:08] well by they you mean me :P [23:29:30] lol [23:29:34] i did not actually see the name [23:29:40] heh [23:29:41] "If you'd like to, we can start a slack channel for the developers and the community." [23:29:55] community = paladox (too) [23:30:04] yeh [23:30:17] devs: "ok, but we are busy dev-ing" [23:30:46] lol [23:30:48] i need to figure out how to add a "metafo" record to DNS [23:30:53] mutante they use gerrit too :P [23:31:08] paladox: 2.16 ? [23:31:14] 2.12 [23:31:39] wow, we are more modern for once [23:31:44] mutante http://cr.skytechnology.pl:8081/#/q/status:open [23:31:46] [ Gerrit Code Review ] - cr.skytechnology.pl:8081 [23:32:52] [ ] install apache2, add reverse proxy config for 443 to 8081 [23:33:00] lol [23:33:01] [ ] add redirect from 80 to 443 [23:33:04] [ ] fix SSL_ERROR_RX_RECORD_TOO_LONG [23:33:23] lol [23:34:15] https://www.thesslstore.com/blog/ssl_error_rx_record_too_long/ [23:34:16] [ How to fix the SSL_ERROR_RX_RECORD_TOO_LONG Firefox Error ] - www.thesslstore.com [23:35:24] paladox: i need the "metafo" record mentioned in https://wikitech.wikimedia.org/wiki/DNS/Discovery#Active/passive_services :) [23:35:25] [ DNS/Discovery - Wikitech ] - wikitech.wikimedia.org [23:35:42] and i am supposed to read .. ehm... [23:35:58] https://wikitech.wikimedia.org/wiki/Load_Balanced_Services_And_Conftool [23:35:58] [ Load Balanced Services And Conftool - Wikitech ] - wikitech.wikimedia.org [23:36:10] do you have that yet? [23:36:21] oh [23:36:30] good lord, i have no idea how to do metafo :P [23:36:58] "good lord" is a new term you like :) [23:37:06] lol [23:37:34] maybe it is even a typo [23:38:05] ah, no, it's not. there are actually some in the DNS repo. i gotta try copying [23:39:41] mutante seems metafo neeeds confctl? [23:40:06] so the thing is some services are active-active (in both DCs) [23:40:14]