[00:03:02] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.09, 20.72, 13.08 [00:05:07] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.95, 19.69, 13.69 [00:24:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkNe [00:24:13] [02miraheze/puppet] 07paladox 039305bf6 - Update init.pp [00:24:49] [02puppet] 07paladox commented on commit 039305bf6433782986ab0b0f29ff1e8a9a6901301a - 13https://git.io/JJkNf [00:25:46] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:37:12] !log regenerating backups on bacula1 [00:37:16] err bacula2 [00:37:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:17] will fix [00:40:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkNB [00:40:20] [02miraheze/services] 07MirahezeSSLBot 0376babf1 - BOT: Updating services config for wikis [01:11:10] PROBLEM - mw4 Disk Space on mw4 is WARNING: DISK WARNING - free space: / 2016 MB (10% inode=68%); [01:11:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkNH [01:11:12] [02miraheze/puppet] 07paladox 030572b5d - bacula::client: Add port 9103 [01:13:24] RECOVERY - mw4 Disk Space on mw4 is OK: DISK OK - free space: / 6680 MB (35% inode=68%); [01:19:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJkAe [01:19:29] [02miraheze/mw-config] 07paladox 03353a15b - Fix setting wgHiddenPrefs for dcwiki [01:19:30] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:19:32] [02mw-config] 07paladox opened pull request 03#3137: Fix setting wgHiddenPrefs for dcwiki - 13https://git.io/JJkAv [01:19:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJkAf [01:19:52] [02miraheze/mw-config] 07paladox 037879bb6 - Update LocalSettings.php [01:19:53] [02mw-config] 07paladox synchronize pull request 03#3137: Fix setting wgHiddenPrefs for dcwiki - 13https://git.io/JJkAv [01:20:02] [02mw-config] 07paladox closed pull request 03#3137: Fix setting wgHiddenPrefs for dcwiki - 13https://git.io/JJkAv [01:20:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJkAJ [01:20:05] [02miraheze/mw-config] 07paladox 03d7fa2bf - Fix setting wgHiddenPrefs for dcwiki (#3137) * Fix setting wgHiddenPrefs for dcwiki * Update LocalSettings.php [01:20:06] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:20:08] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [01:22:13] PROBLEM - mw5 Disk Space on mw5 is WARNING: DISK WARNING - free space: / 2079 MB (10% inode=68%); [01:23:19] PROBLEM - mw6 Disk Space on mw6 is WARNING: DISK WARNING - free space: / 2072 MB (10% inode=68%); [01:24:32] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.45, 22.53, 19.16 [01:25:34] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/JJkAI [01:25:35] [02miraheze/mw-config] 07paladox 0355b9204 - Revert "Fix setting wgHiddenPrefs for dcwiki (#3137)" This reverts commit d7fa2bf95ff35d9d1a5cef1577f2e257da63b9c2. [01:25:37] [02miraheze/mw-config] 07paladox 036d58ad7 - Fix setting wgHiddenPrefs for dcwiki Change-Id: I2e8b6bbd600ff651ad6d609ffeff629247397779 [01:26:12] RECOVERY - mw5 Disk Space on mw5 is OK: DISK OK - free space: / 7021 MB (36% inode=68%); [01:27:21] RECOVERY - mw6 Disk Space on mw6 is OK: DISK OK - free space: / 7016 MB (36% inode=68%); [01:32:39] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.49, 21.65, 19.94 [01:36:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.30, 23.54, 21.13 [01:40:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkAu [01:40:45] [02miraheze/puppet] 07paladox 033251ca1 - bacula::client: Add port 9101 to firewall [01:42:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJkAg [01:42:13] [02miraheze/puppet] 07paladox 0394c9aeb - bacula: Move port 9101/9103 to director.pp [01:42:14] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [01:42:16] [02puppet] 07paladox opened pull request 03#1423: bacula: Move port 9101/9103 to director.pp - 13https://git.io/JJkA2 [01:42:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJkAw [01:42:54] [02miraheze/puppet] 07paladox 0345bbbe4 - Update director.pp [01:42:55] [02puppet] 07paladox synchronize pull request 03#1423: bacula: Move port 9101/9103 to director.pp - 13https://git.io/JJkA2 [01:42:57] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.55, 23.28, 21.38 [01:43:20] [02puppet] 07paladox closed pull request 03#1423: bacula: Move port 9101/9103 to director.pp - 13https://git.io/JJkA2 [01:43:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJkAr [01:43:23] [02miraheze/puppet] 07paladox 0341615d4 - bacula: Move port 9101/9103 to director.pp (#1423) * bacula: Move port 9101/9103 to director.pp * Update director.pp [01:43:24] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [01:43:26] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [01:44:59] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.91, 21.78, 21.09 [01:45:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkAK [01:45:02] [02miraheze/puppet] 07paladox 03f58a1d9 - fix duplicates [01:55:30] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.23, 18.79, 20.31 [02:16:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.65, 21.07, 20.43 [02:22:26] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.99, 23.46, 21.57 [02:24:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.06, 22.50, 21.45 [02:28:34] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.05, 19.17, 20.34 [02:36:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.94, 21.16, 20.89 [02:38:28] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.33, 21.54, 21.02 [02:44:49] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.61, 22.34, 21.86 [02:46:49] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.11, 23.27, 22.25 [02:48:53] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.85, 22.03, 21.95 [02:55:13] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.06, 24.06, 22.53 [02:57:16] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.61, 22.40, 22.08 [02:59:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.38, 22.59, 22.14 [03:01:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.89, 21.05, 21.59 [03:03:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.37, 22.11, 21.90 [03:07:38] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.60, 22.78, 22.25 [03:15:47] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.40, 18.59, 20.31 [03:28:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.14, 21.02, 20.42 [03:32:38] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.33, 21.67, 20.71 [03:36:53] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.11, 20.80, 20.53 [03:45:06] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.33, 18.95, 20.21 [04:01:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.39, 20.04, 19.81 [04:07:36] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.68, 2.26, 1.37 [04:08:01] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.79, 19.90, 20.06 [04:09:38] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.96, 1.83, 1.32 [04:11:45] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.66, 1.70, 1.33 [04:12:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.04, 22.25, 21.00 [04:16:29] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.67, 19.44, 20.20 [04:20:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.90, 20.62, 20.40 [04:22:43] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.66, 19.23, 19.93 [04:50:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkhj [04:50:23] [02miraheze/services] 07MirahezeSSLBot 03c3d1b20 - BOT: Updating services config for wikis [04:51:02] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.20, 20.91, 19.51 [04:53:15] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.57, 21.56, 20.00 [04:57:23] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.79, 18.29, 19.17 [05:18:51] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.15, 20.49, 19.28 [05:20:58] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.22, 23.28, 20.40 [05:29:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.79, 23.81, 22.09 [05:38:01] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.82, 22.84, 22.15 [05:50:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.22, 21.92, 22.76 [05:52:34] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.21, 24.60, 23.56 [05:54:37] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.26, 21.62, 22.60 [06:03:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.62, 25.36, 23.71 [06:04:13] [02mw-config] 07soratako opened pull request 03#3138: Change license for wikigrowthwiki per T5872 - 13https://git.io/JJIeD [06:33:22] PROBLEM - db7 Current Load on db7 is WARNING: WARNING - load average: 6.93, 6.93, 6.34 [06:39:32] RECOVERY - db7 Current Load on db7 is OK: OK - load average: 5.94, 6.61, 6.42 [06:45:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIvg [06:45:17] [02miraheze/services] 07MirahezeSSLBot 0300cc45d - BOT: Updating services config for wikis [06:47:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.67, 22.06, 23.88 [06:55:53] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 42.08, 27.22, 24.77 [06:59:47] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.10, 6.34, 4.16 [07:01:55] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.89, 6.78, 4.60 [07:03:59] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 4.07, 5.95, 4.58 [07:14:22] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=reallifevillainswiki /home/reception/reallifevillains_pages_full.xml [07:14:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:17:54] [02mw-config] 07Reception123 closed pull request 03#3138: Change license for wikigrowthwiki per T5872 - 13https://git.io/JJIeD [07:17:56] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIfc [07:17:57] [02miraheze/mw-config] 07soratako 032267ad3 - Change license for wikigrowthwiki per T5872 (#3138) [07:22:19] RECOVERY - bacula2 Bacula Databases db7 on bacula2 is OK: OK: Full, 22602 files, 80.86GB, 2020-07-05 07:21:00 (1.3 minutes ago) [07:25:13] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 13.94, 20.30, 23.95 [07:27:17] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.32, 22.05, 24.13 [07:35:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.00, 22.80, 23.84 [07:48:24] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.78, 1.40, 1.06 [07:50:28] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.13, 1.28, 1.06 [08:04:58] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.91, 24.06, 22.30 [08:07:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.11, 22.86, 22.07 [08:19:24] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.02, 21.93, 21.36 [08:21:31] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.09, 20.66, 20.95 [08:42:18] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.32, 21.43, 21.03 [08:44:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.77, 22.25, 21.41 [08:46:24] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.12, 23.06, 21.81 [08:48:28] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.11, 21.05, 21.20 [08:50:37] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.59, 22.91, 21.86 [08:52:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.86, 20.95, 21.28 [08:58:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.86, 18.51, 20.08 [09:03:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.00, 21.64, 20.99 [09:09:40] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.66, 18.01, 19.84 [09:13:55] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.39, 20.45, 20.64 [09:15:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.89, 17.68, 19.60 [10:21:29] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.57, 19.99, 18.74 [10:23:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.97, 22.20, 19.71 [10:25:42] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.61, 22.04, 19.96 [10:34:05] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.88, 18.17, 19.22 [11:03:52] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.07, 23.69, 20.06 [11:08:09] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.65, 21.93, 20.17 [11:12:25] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.63, 23.90, 21.11 [11:14:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.09, 21.86, 20.70 [11:20:42] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.94, 19.03, 19.96 [11:37:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.72, 21.21, 19.95 [11:39:29] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.74, 19.46, 19.45 [11:55:00] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.87, 21.12, 19.69 [11:57:06] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.95, 19.19, 19.18 [12:05:42] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.32, 23.11, 21.09 [12:07:45] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.93, 20.24, 20.30 [12:28:51] RECOVERY - bacula2 Bacula Databases db9 on bacula2 is OK: OK: Full, 1047931 files, 30.18GB, 2020-07-05 12:27:00 (1.9 minutes ago) [12:30:50] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 4364 files, 12.72MB, 2020-07-05 12:29:00 (1.8 minutes ago) [12:34:54] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.09, 20.41, 19.78 [12:37:08] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.38, 23.69, 21.12 [12:45:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.39, 22.67, 22.02 [12:46:46] paladox: what's going on with load as well [12:48:21] Backups are currently running [12:49:12] Ok [12:55:12] RECOVERY - bacula2 Bacula Phabricator Static on bacula2 is OK: OK: Full, 83274 files, 3.543GB, 2020-07-05 12:54:00 (1.2 minutes ago) [13:02:02] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.39, 24.10, 22.79 [13:04:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.43, 23.54, 22.77 [13:06:36] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.54, 23.77, 22.90 [13:08:18] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.39, 22.69, 22.61 [13:12:29] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.04, 22.67, 22.45 [13:14:33] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.68, 22.57, 22.47 [13:20:54] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.82, 23.67, 22.67 [13:25:51] [02miraheze/MirahezeBots] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIY9 [13:25:52] [02miraheze/MirahezeBots] 07Pix1234 03505c270 - ZppixBot -> MirahezeBot, update URLs [13:26:11] PROBLEM - gluster1 Current Load on gluster1 is WARNING: WARNING - load average: 7.16, 6.84, 5.24 [13:26:23] [02miraheze/MirahezeBots] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIYQ [13:26:24] [02miraheze/MirahezeBots] 07Pix1234 03a7b84c5 - Missed a instance of “ZppixBot” [13:26:28] PROBLEM - db7 Current Load on db7 is WARNING: WARNING - load average: 7.72, 6.55, 5.88 [13:27:56] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 5.87, 6.33, 5.24 [13:28:39] RECOVERY - db7 Current Load on db7 is OK: OK - load average: 5.46, 6.06, 5.78 [13:32:29] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CRITICAL - load average: 8.95, 8.01, 6.22 [13:34:25] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 3.72, 6.38, 5.84 [13:46:47] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIO2 [13:46:49] [02miraheze/dns] 07paladox 03a9f23ad - Add cloud3 to dns [13:53:14] > [miraheze/dns] paladox pushed 1 commit to master [+0/-0/±1] https://git.io/JJIO2 @Miraheze-IRC haha, cool, I never thought of hosting and publishing DNS zone files on GitHub. A GitHub commit that's within my sphere of competence. 😛 [13:53:16] [ Comparing 93a2d2c429ee...a9f23ad02298 · miraheze/dns · GitHub ] - git.io [14:40:27] PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bluepageswiki.org' expires in 15 day(s) (Tue 21 Jul 2020 14:36:52 GMT +0000). [14:41:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIsa [14:41:14] [02miraheze/ssl] 07MirahezeSSLBot 03ae65c6d - Bot: Update SSL cert for www.bluepageswiki.org [14:42:12] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Tue 21 Jul 2020 14:38:55 GMT +0000). [14:45:00] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.70, 21.58, 23.68 [14:50:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIsj [14:50:23] [02miraheze/ssl] 07MirahezeSSLBot 03061111d - Bot: Update SSL cert for wiki.mobilityengineer.com [14:51:12] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.69, 23.84, 23.97 [14:52:58] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Sat 03 Oct 2020 13:41:06 GMT +0000. [14:58:29] > @Miraheze-IRC haha, cool, I never thought of hosting and publishing DNS zone files on GitHub. A GitHub commit that's within my sphere of competence. 😛 @Doug: Anything we can make public, we do. [14:59:44] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.42, 22.54, 23.92 [15:01:53] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.63, 23.65, 24.16 [15:02:22] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Sat 03 Oct 2020 13:50:15 GMT +0000. [15:02:30] Hey [15:03:34] Question: In wiki requests, if is added comment it is redirected to requester via echo/mail? [15:04:50] No [15:07:11] Ok [15:07:34] So i must do it on user talk page [15:07:36] Thanks [15:08:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.31, 22.63, 23.72 [15:10:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 29.28, 24.76, 24.33 [15:15:23] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.42, 23.21, 23.85 [15:16:53] PROBLEM - augovwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'augovwiki.com' expires in 15 day(s) (Tue 21 Jul 2020 15:12:32 GMT +0000). [15:17:30] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 31.55, 25.57, 24.55 [15:23:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIZG [15:23:14] [02miraheze/ssl] 07MirahezeSSLBot 033130bcc - Bot: Update SSL cert for augovwiki.com [15:26:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.06, 23.12, 23.92 [15:33:18] RECOVERY - augovwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'augovwiki.com' will expire on Sat 03 Oct 2020 14:23:05 GMT +0000. [15:34:42] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.68, 23.43, 23.28 [15:36:51] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.84, 23.12, 23.20 [15:45:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JJInf [15:45:50] [02miraheze/puppet] 07paladox 032d53e60 - Create cloud3.yaml [15:45:52] [02puppet] 07paladox synchronize pull request 03#1226: Introduce cloud role - 13https://git.io/JvWWx [15:47:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.48, 22.08, 22.22 [15:49:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.85, 21.73, 22.06 [15:51:31] [02puppet] 07paladox synchronize pull request 03#1226: Introduce cloud role - 13https://git.io/JvWWx [15:51:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJInt [15:51:34] [02miraheze/puppet] 07paladox 03d63bf9d - Update cloud.pp [15:52:30] [02puppet] 07paladox synchronize pull request 03#1226: Introduce cloud role - 13https://git.io/JvWWx [15:52:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJIn3 [15:52:33] [02miraheze/puppet] 07paladox 03bb3a8c3 - Update cloud.pp [15:52:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJInn [15:52:53] [02miraheze/puppet] 07paladox 039a4850c - Update site.pp [15:52:55] [02puppet] 07paladox synchronize pull request 03#1226: Introduce cloud role - 13https://git.io/JvWWx [15:53:09] [02puppet] 07paladox closed pull request 03#1226: Introduce cloud role - 13https://git.io/JvWWx [15:53:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+7/-0/±3] 13https://git.io/JJInC [15:53:12] [02miraheze/puppet] 07paladox 035e0f6ea - Introduce cloud role (#1226) [15:53:14] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:53:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [15:54:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJInR [15:54:34] [02miraheze/puppet] 07paladox 0398d4de7 - fix syntax [15:55:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJInz [15:55:04] [02miraheze/puppet] 07paladox 03a8f1871 - Update init.pp [15:55:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIn2 [15:55:51] [02miraheze/puppet] 07paladox 0390faf72 - Fix path to file [15:57:53] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.22, 22.34, 22.17 [15:58:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJIno [15:58:48] [02miraheze/puppet] 07paladox 03fab142a - Fix [15:59:00] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:54] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.66, 20.96, 21.67 [16:01:00] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:12:37] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.55, 22.62, 21.68 [16:14:43] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.09, 21.74, 21.49 [16:23:07] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.32, 18.92, 20.36 [16:31:52] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.36, 22.39, 20.95 [16:36:01] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.95, 22.40, 21.53 [16:42:27] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.48, 17.84, 19.94 [17:32:11] Hello wpirc40! If you have any questions, feel free to ask and someone should answer soon. [18:03:49] hi [18:08:59] https://meta.miraheze.org/wiki/Requests_for_Comment/Code_of_Conduct_Commission_reform [18:09:00] [ Requests for Comment/Code of Conduct Commission reform - Miraheze Meta ] - meta.miraheze.org [18:09:19] I wish people have made something about the voting methods too [18:09:36] (ref https://meta.miraheze.org/wiki/Talk:Code_of_Conduct/Commission/Election/2019 [18:09:37] [ Talk:Code of Conduct/Commission/Election/2019 - Miraheze Meta ] - meta.miraheze.org [18:14:02] Well, it might be a bit late for adding a section, but there's very little stopping you from making a new proposal [18:14:30] but I guess that just wasn't as much of a concern overall compared to the functionality of the commission itself [18:25:57] wpirc40: how can we help [18:26:23] RhinosF1 Hello, i need to run bot in dump of eswiki in my own MediaWiki. [18:35:56] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.27, 19.64, 17.32 [18:38:04] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.21, 18.56, 17.24 [18:52:58] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.22, 21.06, 19.38 [18:57:03] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.50, 17.76, 18.49 [19:17:15] [02puppet] 07JohnFLewis commented on commit 035e0f6eab95a2f63e627639034c874ea3c54ee959 - 13https://git.io/JJI8f [19:18:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.37, 20.39, 18.45 [19:20:17] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 20.03, 19.40, 18.29 [19:27:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJI8u [19:27:54] [02miraheze/puppet] 07paladox 03877bf36 - Cloud: Add ssh::server::permit_root [19:27:55] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:27:57] [02puppet] 07paladox opened pull request 03#1424: Cloud: Add ssh::server::permit_root - 13https://git.io/JJI8z [19:28:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJI8g [19:28:18] [02miraheze/puppet] 07paladox 03adf9bac - Update cloud2.yaml [19:28:20] [02puppet] 07paladox synchronize pull request 03#1424: Cloud: Add ssh::server::permit_root - 13https://git.io/JJI8z [19:28:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJI8r [19:28:42] [02miraheze/puppet] 07paladox 03f39467c - Update cloud3.yaml [19:28:44] [02puppet] 07paladox synchronize pull request 03#1424: Cloud: Add ssh::server::permit_root - 13https://git.io/JJI8z [19:30:33] [02puppet] 07paladox commented on commit 035e0f6eab95a2f63e627639034c874ea3c54ee959 - 13https://git.io/JJI8i [19:31:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJI8P [19:31:14] [02miraheze/puppet] 07paladox 035a67ee8 - cloud: Fix hiera values [19:31:16] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [19:31:17] [02puppet] 07paladox opened pull request 03#1425: cloud: Fix hiera values - 13https://git.io/JJI8X [19:31:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJI81 [19:31:49] [02miraheze/puppet] 07paladox 032cc30f0 - Update cloud1.yaml [19:31:51] [02puppet] 07paladox synchronize pull request 03#1425: cloud: Fix hiera values - 13https://git.io/JJI8X [19:32:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJI8M [19:32:12] [02miraheze/puppet] 07paladox 0343a45b1 - Update cloud2.yaml [19:32:13] [02puppet] 07paladox synchronize pull request 03#1425: cloud: Fix hiera values - 13https://git.io/JJI8X [19:32:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJI8D [19:32:37] [02miraheze/puppet] 07paladox 03726d6e2 - Update cloud3.yaml [19:32:38] [02puppet] 07paladox synchronize pull request 03#1425: cloud: Fix hiera values - 13https://git.io/JJI8X [19:32:54] [02puppet] 07paladox closed pull request 03#1425: cloud: Fix hiera values - 13https://git.io/JJI8X [19:32:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JJI8y [19:32:57] [02miraheze/puppet] 07paladox 03efeaabb - cloud: Fix hiera values (#1425) [19:32:58] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [19:33:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [19:42:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.15, 22.79, 19.98 [19:42:32] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:43:22] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:44:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJI43 [19:44:11] [02miraheze/puppet] 07paladox 03514573a - Fix [19:44:13] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [19:44:14] [02puppet] 07paladox opened pull request 03#1426: Fix - 13https://git.io/JJI4s [19:44:36] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.39, 18.64, 18.80 [19:45:07] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:45:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJI4Z [19:45:14] [02miraheze/services] 07MirahezeSSLBot 03104d4d3 - BOT: Updating services config for wikis [19:45:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJI4n [19:45:18] [02miraheze/puppet] 07paladox 03a7ef259 - Update init.pp [19:45:19] [02puppet] 07paladox synchronize pull request 03#1426: Fix - 13https://git.io/JJI4s [19:45:26] [02puppet] 07paladox edited pull request 03#1426: Cloud: Fix hiera values part 2 - 13https://git.io/JJI4s [19:45:36] [02puppet] 07paladox closed pull request 03#1426: Cloud: Fix hiera values part 2 - 13https://git.io/JJI4s [19:45:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJI48 [19:45:39] [02miraheze/puppet] 07paladox 0386d1960 - Cloud: Fix hiera values part 2 (#1426) * Fix * Update init.pp [19:46:02] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [19:46:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [19:46:41] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:53:36] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:53:41] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:54:17] [02miraheze/bots-web] 07RhinosF1 pushed 031 commit to 03scfix [+0/-0/±1] 13https://git.io/JJI4M [19:54:19] [02miraheze/bots-web] 07RhinosF1 03f9778ad - Allow css indexing [19:54:20] [02bots-web] 07RhinosF1 created branch 03scfix - 13https://git.io/JJfyu [19:54:22] [02bots-web] 07RhinosF1 opened pull request 03#12: Allow css indexing - 13https://git.io/JJI4S [19:54:43] [02bots-web] 07RhinosF1 edited pull request 03#12: Robot fixes - 13https://git.io/JJI4S [19:56:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.37, 22.60, 20.25 [19:58:25] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.96, 18.00, 18.85 [19:59:25] [02miraheze/bots-web] 07RhinosF1 pushed 031 commit to 03scfix [+0/-0/±1] 13https://git.io/JJI4j [19:59:27] [02miraheze/bots-web] 07RhinosF1 0343636bd - Update sitemap.xml [19:59:28] [02bots-web] 07RhinosF1 synchronize pull request 03#12: Robot fixes - 13https://git.io/JJI4S [20:04:37] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:39] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 20 Aug 2020 14:54:45 GMT +0000. [20:22:35] [02bots-web] 07MacFan4000 closed pull request 03#12: Robot fixes - 13https://git.io/JJI4S [20:22:37] [02miraheze/bots-web] 07MacFan4000 pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/JJIRe [20:22:38] [02miraheze/bots-web] 07MacFan4000 03b581f86 - Merge pull request #12 from miraheze/scfix Robot fixes [20:22:41] [02bots-web] 07MacFan4000 deleted branch 03scfix - 13https://git.io/JJfyu [20:22:43] [02miraheze/bots-web] 07MacFan4000 deleted branch 03scfix [20:24:57] [02miraheze/bots-web] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://git.io/JJIRt [20:24:58] [02miraheze/bots-web] 07MacFan4000 03217198b - Try something with travis [20:25:00] [02bots-web] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://git.io/JJfyu [20:25:01] [02bots-web] 07MacFan4000 opened pull request 03#13: Try something with travis - 13https://git.io/JJIRq [20:34:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [20:34:53] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:35:18] [02bots-web] 07MacFan4000 closed pull request 03#13: Try something with travis - 13https://git.io/JJIRq [20:35:19] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.758 second response time [20:35:22] [02miraheze/bots-web] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [20:35:24] [02bots-web] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://git.io/JJfyu [20:35:29] we may be down right now [20:35:41] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw6 mw7 [20:35:57] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 50% [20:36:00] 502 [20:36:01] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw4 mw6 mw7 [20:36:12] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.241 second response time [20:36:26] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.002 second response time [20:36:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [20:36:48] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:37:10] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [20:37:10] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:37:20] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:37:29] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [20:37:48] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15533 bytes in 1.003 second response time [20:38:26] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [20:39:49] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: HTTP CRITICAL - No data received from host [20:41:22] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.314 second response time [20:41:25] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.005 second response time [20:41:50] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 1.002 second response time [20:42:05] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15533 bytes in 0.038 second response time [20:42:05] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15533 bytes in 0.014 second response time [20:42:13] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.346 second response time [20:43:02] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:27] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:50] yay everyone is back. Re-pinging RhinosF1. 502/503s everywhere on Miraheze :( [20:43:57] !log MariaDB [(none)]> set global table_definition_cache = 5000; - db9 [20:44:15] !log MariaDB [(none)]> set global table_open_cache = 5000; - db9 [20:45:02] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15539 bytes in 1.008 second response time [20:45:12] musikanimal, known, paladox is looking into it [20:45:18] freenode is obviously problems too which isn't helping... [20:45:19] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.003 second response time [20:45:33] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 39% [20:45:34] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [20:45:46] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15539 bytes in 0.003 second response time [20:45:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:46:05] ok thanks! [20:46:21] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [20:46:34] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [20:46:59] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [20:48:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:53:34] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.01, 18.45, 14.87 [20:56:00] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 9.61, 15.26, 14.24 [20:56:07] musikanimal fixed [20:56:14] many thanks! [20:59:10] PROBLEM - db9 Current Load on db9 is CRITICAL: CRITICAL - load average: 19.60, 29.15, 14.68 [21:35:29] PROBLEM - db9 Current Load on db9 is WARNING: WARNING - load average: 1.54, 2.25, 7.52 [21:37:26]