[00:22:14] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.07, 6.25, 5.84 [00:22:51] !log depool mw2 and reboot [00:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:25:53] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:05] PROBLEM - mw2 php-fpm on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:26:37] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [00:26:42] PROBLEM - mw2 SSH on mw2 is CRITICAL: connect to address 185.52.2.113 and port 22: Connection refused [00:27:15] !log root@mw1:/home/reception# tar -zcvf wikibackups.tar.gz wikibackups [00:27:16] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [00:27:18] PROBLEM - mw2 Puppet on mw2 is CRITICAL: connect to address 185.52.2.113 port 5666: Connection refusedconnect to host 185.52.2.113 port 5666: Connection refused [00:27:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:27:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [00:27:43] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [00:28:15] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 0.27, 0.06, 0.02 [00:28:27] !log repool mw2 [00:28:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:28:39] RECOVERY - mw2 SSH on mw2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [00:29:11] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 38420 MB (50% inode=99%); [00:29:13] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [00:29:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:29:43] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:29:48] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.007 second response time [00:29:54] RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 13 processes with command name 'php-fpm7.2' [00:30:37] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:39:05] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 703 bytes in 0.085 second response time [00:41:02] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.399 second response time [00:44:27] Hi! Here is the list of currently open high priority tasks on Phabricator [00:44:34] No updates for 6 days - https://phabricator.miraheze.org/T4514 - It is garbled (Japanese:文字化けが発生しています。 - authored by hiyuame0523, assigned to None [00:44:40] No updates for 3 days - https://phabricator.miraheze.org/T4445 - Update PDF Rendering for nenawiki.org and ndg.nenawiki.org - authored by MikeV, assigned to None [00:59:10] !log root@mw1:/home/reception# rm -rf wikibackups [00:59:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:02:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPRi [01:02:17] [02miraheze/mw-config] 07paladox 034e23db7 - Revert "Sitenotice for the enabling of ElasticSearch on all wikis" This reverts commit 97678dc924ee4579a356eb5a412177eec99d9c60. [01:04:48] !log depool mw2 [01:04:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:13:45] !log repool mw2 [01:13:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:15:27] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fjPRQ [01:15:28] [02miraheze/mw-config] 07paladox 0301d3623 - Revert "Set wmgUseCirrusSearch to true" This reverts commit 82db9bf1479d30d0324591f8867b19d1d34873e1. Reason for revert: Change-Id: I00b9f4a57ebe3384136d9e01bd49fce837a3ca44 [01:15:30] [02miraheze/mw-config] 07paladox 0395b29be - Merge "Revert "Set wmgUseCirrusSearch to true"" [12:10:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPwd [12:10:15] [02miraheze/services] 07MirahezeSSLBot 032b90b62 - BOT: Updating services config for wikis [12:28:15] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 52180 MB (10% inode=99%); [14:37:42] RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 4375781 files, 364.8GB, 2019-07-11 14:36:00 (1.7 minutes ago) [14:45:27] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.73, 2.07, 1.04 [14:49:28] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.49, 1.90, 1.23 [14:53:27] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.51, 2.29, 1.54 [14:55:26] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.32, 1.93, 1.50 [14:57:26] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.09, 1.63, 1.44 [15:03:26] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.44, 1.92, 1.59 [15:05:26] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.37, 1.72, 1.56 [15:07:26] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.14, 1.51, 1.50 [15:25:26] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.46, 1.85, 1.63 [15:27:26] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.14, 1.58, 1.56 [15:35:26] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.74, 1.88, 1.68 [15:36:13] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPie [15:36:14] [02miraheze/ManageWiki] 07translatewiki 03ffe891d - Localisation updates from https://translatewiki.net. [15:36:16] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPiv [15:36:17] [02miraheze/MirahezeMagic] 07translatewiki 030299357 - Localisation updates from https://translatewiki.net. [15:37:26] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.18, 1.63, 1.61 [15:55:27] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.96, 1.83, 1.67 [15:57:27] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.13, 1.90, 1.72 [15:59:26] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.48, 1.36, 1.54 [16:07:47] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 80047 files, 2.255GB, 2019-07-11 16:06:00 (1.8 minutes ago) [16:27:48] PROBLEM - knuxwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:43] RECOVERY - knuxwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'knuxwiki.com' will expire on Fri 12 Jun 2020 11:59:59 PM GMT +0000. [16:52:56] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.18, 6.50, 4.91 [16:54:53] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.63, 6.34, 5.03 [17:05:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPP5 [17:05:14] [02miraheze/services] 07MirahezeSSLBot 03450000a - BOT: Updating services config for wikis [17:06:35] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.01, 7.04, 5.98 [17:08:32] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.74, 6.47, 5.90 [17:12:34] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.67, 7.59, 6.46 [17:18:29] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.34, 7.60, 6.92 [17:20:27] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.39, 7.71, 7.02 [17:22:26] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.46, 7.34, 6.97 [17:30:19] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.00, 5.99, 6.56 [17:44:10] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.61, 6.32, 5.39 [17:46:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.99, 6.34, 5.52 [18:36:10] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.52, 6.66, 5.41 [18:42:10] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.74, 7.04, 6.03 [18:46:10] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.29, 7.48, 6.42 [18:48:10] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.01, 6.85, 6.31 [18:50:10] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.51, 6.57, 6.26 [18:56:10] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.18, 7.32, 6.68 [19:00:10] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.52, 6.73, 6.57 [19:06:10] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.18, 7.97, 7.11 [19:08:11] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.53, 7.61, 7.08 [19:20:14] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.29, 7.70, 7.17 [19:23:23] paladox, ^ [19:23:43] * paladox looks [19:24:12] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.59, 7.75, 7.31 [19:28:09] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.48, 6.00, 6.70 [19:38:46] !log depool mw2 [19:38:50] Hello Einstein! If you have any questions feel free to ask and someone should answer soon. [19:38:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:39:37] I wanted to know, if there is a possibility to change the domain? [19:41:06] PROBLEM - lizardfs2 Puppet on lizardfs2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [19:41:09] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [19:41:13] PROBLEM - puppet1 Puppet on puppet1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [19:41:30] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [19:41:39] Einstein hi, yes! [19:41:57] PROBLEM - misc1 Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [19:42:03] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [19:42:06] PROBLEM - bacula1 Puppet on bacula1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [19:42:08] !log repool mw2 [19:42:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.45, 6.33, 5.45 [19:42:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:42:16] Thank you very much for the fast reply. [19:42:18] PROBLEM - lizardfs1 Puppet on lizardfs1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [19:42:18] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [19:42:27] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [19:42:34] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [19:42:37] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [19:42:58] PROBLEM - lizardfs3 Puppet on lizardfs3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [19:43:00] What does this: "!log repool mw2" mean? and how can I change the domain? [19:43:00] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [19:43:07] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [19:43:08] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [19:44:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.85, 5.87, 5.39 [19:46:30] How can I change the domain? [19:48:59] Einstein we can rename wiki's/or you can buy a custom domain, see https://meta.miraheze.org/wiki/Custom_domains (that's for custom domains, to rename your wiki please file a request on https://phabricator.miraheze.org) [19:49:00] [ Custom domains - Miraheze Meta ] - meta.miraheze.org [19:50:27] Thank you very much. I thought so, but I was not sure. Thank you also for the links. [19:51:57] Have all a good afternoon. [20:19:16] Can one of you sysadmins do https://phabricator.miraheze.org/T4535? [20:19:16] [ ⚓ T4535 Request domain to be changed ] - phabricator.miraheze.org [20:20:46] Paladox: ^ [20:21:39] I carn't at the moment. [20:23:24] Paladox: Np [20:40:59] @Void or another meta admin - check rc on meta - someones creating loads of similar named accounts [20:42:36] usually comes out to being nothing [20:42:41] will keep an eye out [20:42:53] voidwalker: Thx [20:50:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPDz [20:50:30] [02miraheze/mw-config] 07paladox 039693d19 - Sitenotice for ElasticSearch maintenance Change-Id: I0f9180e4fae2f84bedac02de6c4f29b7b36036b7 [20:57:09] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:57:12] !log depool mw1 && hack php-fpm childs && restart php-fpm && repool mw1 [20:57:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:59:42] !log depool mw3 && hack php-fpm childs && restart php-fpm && repool mw3 [20:59:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:00:34] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [21:07:48] !log kill $(ps -A -ostat,ppid | awk '/[zZ]/ && !a[$2]++ {print $2}') on mw3 [21:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:11:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjPDX [21:11:57] [02miraheze/puppet] 07paladox 03ee48ee0 - jobrunner: Use flock when running RottenLink maintenance script [21:11:59] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:12:00] [02puppet] 07paladox opened pull request 03#1026: jobrunner: Use flock when running RottenLink maintenance script - 13https://git.io/fjPD1 [21:24:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPDQ [21:24:37] [02miraheze/mw-config] 07paladox 038ace7e1 - extend maintenance window Change-Id: Ib2ac6a9626f62cf6b1fd58a295f7aaa2eabb12d5 [21:32:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjPDb [21:32:22] [02miraheze/puppet] 07paladox 0309947f5 - Create fileLockScript.sh [21:32:24] [02puppet] 07paladox synchronize pull request 03#1026: jobrunner: Use flock when running RottenLink maintenance script - 13https://git.io/fjPD1 [21:32:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjPDA [21:32:57] [02miraheze/puppet] 07paladox 03909e7b4 - Update init.pp [21:32:59] [02puppet] 07paladox synchronize pull request 03#1026: jobrunner: Use flock when running RottenLink maintenance script - 13https://git.io/fjPD1 [21:33:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjPDp [21:33:34] [02miraheze/puppet] 07paladox 03578c0d5 - Update jobrunner.pp [21:33:35] [02puppet] 07paladox synchronize pull request 03#1026: jobrunner: Use flock when running RottenLink maintenance script - 13https://git.io/fjPD1 [21:33:57] [02puppet] 07paladox edited pull request 03#1026: jobrunner: Add new script that file locks (and convert RottenLinks script to use it) - 13https://git.io/fjPD1 [21:37:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjPye [21:37:12] [02miraheze/puppet] 07paladox 039ab0aab - Update fileLockScript.sh [21:37:14] [02puppet] 07paladox synchronize pull request 03#1026: jobrunner: Add new script that file locks (and convert RottenLinks script to use it) - 13https://git.io/fjPD1 [21:37:35] [02puppet] 07paladox closed pull request 03#1026: jobrunner: Add new script that file locks (and convert RottenLinks script to use it) - 13https://git.io/fjPD1 [21:37:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/fjPyv [21:37:38] [02miraheze/puppet] 07paladox 03329ae28 - jobrunner: Add new script that file locks (and convert RottenLinks script to use it) (#1026) * jobrunner: Use flock when running RottenLink maintenance script * Create fileLockScript.sh * Update init.pp * Update jobrunner.pp * Update fileLockScript.sh [21:39:54] !log kill -9 7625 (mw3) [21:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:40:02] !log kill -9 7626 (mw3) [21:40:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:40:19] !log kill -9 749 (mw3) [21:40:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:41:50] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:41:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:50:03] !log depool mw3, and reboot [21:50:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:51:15] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:18] PROBLEM - elasticsearch1 Disk Space on elasticsearch1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:45] PROBLEM - elasticsearch1 SSH on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:40] PROBLEM - Host elasticsearch1 is DOWN: PING CRITICAL - Packet loss = 100% [21:52:56] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:43] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:53:48] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:55] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:53:58] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:37] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [21:54:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:49] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:58] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:00] PROBLEM - mw3 SSH on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:10] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:16] PROBLEM - Host mw3 is DOWN: PING CRITICAL - Packet loss = 100% [21:55:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:56:39] !log repool mw3 [21:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:57:06] RECOVERY - Host elasticsearch1 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms [21:57:19] RECOVERY - Host mw3 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [21:57:25] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.017 second response time [21:57:32] RECOVERY - elasticsearch1 Disk Space on elasticsearch1 is OK: DISK OK - free space: / 62609 MB (76% inode=99%); [21:57:44] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [21:57:44] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 34903 MB (45% inode=99%); [21:57:55] RECOVERY - elasticsearch1 SSH on elasticsearch1 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [21:57:57] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 5 processes with command name 'php-fpm7.2' [21:58:37] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:58:42] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.006 second response time [21:59:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:59:43] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:30:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPyP [22:30:14] [02miraheze/services] 07MirahezeSSLBot 035d8bf8f - BOT: Updating services config for wikis [22:34:34] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [22:40:34] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:45:16] !log reverting php-fpm childs on mw* [22:45:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:51:03] paladox: whatever quircwiki is on seems slow [22:51:22] seems fast to me [22:51:40] paladox: wasn't a few minutes a go [22:52:01] hmm, must have been when i was switching the childs back to what they were before [22:52:32] paladox: 23:44 first stopped and was working until 23:48 [22:52:53] hmm, that was a minute after i started it [22:53:08] paladox: maybe that then [22:53:31] oh, i ment before not after sorry [22:53:46] paladox: strange [22:53:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPyF [22:53:59] [02miraheze/mw-config] 07paladox 034a05099 - Set wmgUseCirrusSearch to true Change-Id: I896833bc1adea025b13446a6bbb80e7f7fdd6dfe [22:54:20] paladox: seems fine now anyway [22:54:25] ok [22:58:11] !log running updateOneSearchIndexConfig.php for all wikis [22:58:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:10:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.26, 6.70, 5.61 [23:14:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.69, 7.99, 6.44 [23:20:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.39, 7.63, 6.72 [23:24:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.31, 7.11, 6.78 [23:26:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.95, 6.14, 6.48