[00:08:17] [02mw-config] 07paladox closed pull request 03#2798: Make all wikis that start with the letter a use the new mount - 13https://git.io/JerR5 [00:08:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jer0G [00:08:20] [02miraheze/mw-config] 07paladox 038a65ff5 - Make all wikis that start with the letter a use the new mount (#2798) * Make all wikis that start with the letter a use the new mount * Update LocalSettings.php * Update LocalSettings.php * Update LocalSettings.php [00:10:47] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:10:49] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [01:19:33] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerEn [01:19:34] [02miraheze/MatomoAnalytics] 07paladox 037314bfb - Fix loading minified js Based on https://github.com/matomo-org/matomo/blob/master/js/README.md#why-use-jsindexphp we need to use js/index.php [01:19:35] [ matomo/README.md at master · matomo-org/matomo · GitHub ] - github.com [01:20:14] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JerEc [01:20:15] [02miraheze/mediawiki] 07paladox 03158e1c1 - Update MA [01:21:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerEW [01:21:27] [02miraheze/puppet] 07paladox 032299f12 - Update default.vcl [01:24:21] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerEl [01:24:22] [02miraheze/MatomoAnalytics] 07paladox 0380fd28f - Update MatomoAnalyticsHooks.php [01:25:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JerE4 [01:25:06] [02miraheze/mediawiki] 07paladox 03395e7a1 - Update MA [01:27:10] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.30, 13.99, 11.28 [01:28:12] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JerEu [01:28:13] paladox: Do you see a mistake? https://commons.miraheze.org/wiki/Talk:Main_Page [01:28:14] [02miraheze/mediawiki] 07paladox 0377fc819 - Update MA [01:28:15] [ Unknown topic - Miraheze Commons ] - commons.miraheze.org [01:29:20] hum, then it won't be me? [01:30:14] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:31:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.40, 14.70, 12.17 [01:34:20] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:34:35] [02miraheze/MatomoAnalytics] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/JerEV [01:34:36] [02miraheze/MatomoAnalytics] 07paladox 037bb5899 - Revert "Update MatomoAnalyticsHooks.php" This reverts commit 80fd28f28b2215619467b13aa7c99903db41a565. [01:34:38] [02miraheze/MatomoAnalytics] 07paladox 032da22e1 - Revert "Fix loading minified js" This reverts commit 7314bfb629e6c091b1b3eb6fdbb24b858e560972. [01:34:39] [02miraheze/MatomoAnalytics] 07paladox 03ee3716a - Revert "Load piwik.min.js instead of piwik.js" This reverts commit a771a7f00cee45639f93b530d869f051e70adfeb. [01:35:39] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JerEw [01:35:41] [02miraheze/mediawiki] 07paladox 0376d63b9 - Update MA [01:36:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerEo [01:36:13] [02miraheze/puppet] 07paladox 039a40208 - Update default.vcl [01:37:50] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JerE6 [01:37:52] [02miraheze/mediawiki] 07paladox 03c25a69c - Update MA [01:39:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.41, 14.25, 12.82 [01:41:47] hispano76 i've deleted the page. [01:41:50] *now [01:41:59] should be fixed (to create new discussions)? [01:43:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.81, 15.24, 13.58 [01:53:13] paladox: pm? [01:53:22] yup, sure. [01:55:14] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 10.65, 12.94, 13.38 [01:59:12] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.23, 14.88, 13.93 [02:03:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.49, 15.46, 14.42 [02:03:30] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.40, 3.65, 3.18 [02:05:13] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.02, 15.64, 14.57 [02:05:31] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.50, 3.52, 3.18 [02:07:32] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.81, 3.20, 3.10 [02:13:12] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 10.33, 15.40, 15.26 [02:21:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.48, 15.86, 15.23 [02:25:15] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.63, 15.14, 15.07 [02:27:15] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.99, 16.05, 15.37 [02:29:11] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.54, 15.59, 15.28 [02:31:11] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.44, 16.09, 15.46 [02:33:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.50, 14.99, 15.12 [02:33:11] PROBLEM - sahitya.shaunak.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sahitya.shaunak.in' expires in 15 day(s) (Sat 30 Nov 2019 02:30:33 AM GMT +0000). [02:33:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeruo [02:33:26] [02miraheze/ssl] 07MirahezeSSLBot 0358cd479 - Bot: Update SSL cert for sahitya.shaunak.in [02:35:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.60, 15.20, 15.16 [02:43:09] RECOVERY - sahitya.shaunak.in - LetsEncrypt on sslhost is OK: OK - Certificate 'sahitya.shaunak.in' will expire on Wed 12 Feb 2020 01:33:18 AM GMT +0000. [03:13:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.10, 14.15, 15.75 [03:17:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.48, 15.31, 15.78 [03:19:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.88, 14.63, 15.48 [03:21:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.78, 14.95, 15.46 [03:23:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.89, 13.87, 15.02 [03:29:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.13, 13.98, 14.60 [03:31:09] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.20, 13.38, 14.29 [03:37:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.36, 14.37, 14.42 [03:39:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.59, 14.20, 14.32 [03:43:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.19, 15.23, 14.71 [03:47:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.54, 15.33, 14.87 [03:51:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.10, 16.07, 15.30 [03:53:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.65, 14.89, 14.92 [03:57:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.64, 14.95, 14.83 [03:59:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.26, 15.06, 14.91 [04:01:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.17, 15.89, 15.23 [04:05:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.24, 15.69, 15.38 [04:07:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.23, 15.34, 15.25 [04:13:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.07, 14.13, 14.88 [04:17:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.29, 15.52, 15.22 [04:19:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 10.42, 14.13, 14.80 [04:21:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.02, 15.47, 15.20 [04:23:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.92, 14.21, 14.74 [04:25:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.92, 14.95, 14.90 [04:27:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 9.61, 13.10, 14.25 [04:33:04] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 10.02, 11.91, 13.37 [04:37:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.15, 13.81, 13.86 [04:43:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.01, 15.27, 14.34 [04:45:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.78, 14.45, 14.14 [04:47:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 10.35, 12.85, 13.58 [04:51:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.52, 13.75, 13.90 [04:53:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 10.89, 12.63, 13.46 [05:03:21] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.85, 3.23, 2.83 [05:07:20] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.25, 3.19, 2.89 [05:25:39] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.61, 14.29, 13.16 [05:33:29] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 10.82, 13.05, 13.26 [06:18:29] !log reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki mashedpotaytoeswiki --r="MediaWiki exception, force delete" --u="Reception123" /home/reception/deletebatch.txt [06:18:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [06:26:32] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2811 MB (11% inode=94%); [06:44:15] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.07, 2.63, 1.44 [06:46:14] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 3.08, 2.92, 1.70 [06:46:25] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:54:25] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:22:46] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.21, 3.40, 3.09 [07:24:45] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.74, 3.14, 3.03 [07:32:42] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.49, 3.27, 3.10 [07:38:42] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.99, 3.26, 3.16 [07:46:45] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.06, 3.76, 3.41 [07:48:45] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.31, 3.58, 3.38 [07:50:48] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.29, 3.77, 3.46 [07:52:50] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.67, 3.71, 3.47 [07:58:59] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.69, 3.21, 3.34 [08:53:23] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [09:03:19] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.72, 3.02, 2.50 [09:05:19] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.33, 3.18, 2.63 [13:19:52] !log restart lizardfs-master on lizardfs6 [13:20:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:48:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JerK9 [13:48:28] [02miraheze/mw-config] 07paladox 0336f7c50 - Move all wikis that start with the letter b to the new mount [13:48:29] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [13:48:31] [02mw-config] 07paladox opened pull request 03#2799: Move all wikis that start with the letter b to the new mount - 13https://git.io/JerKH [13:48:37] [02mw-config] 07paladox closed pull request 03#2799: Move all wikis that start with the letter b to the new mount - 13https://git.io/JerKH [13:48:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerK7 [13:48:40] [02miraheze/mw-config] 07paladox 034b94e7e - Move all wikis that start with the letter b to the new mount (#2799) [13:49:38] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [13:49:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [15:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerPq [15:30:10] [02miraheze/services] 07MirahezeSSLBot 03359f4c4 - BOT: Updating services config for wikis [15:32:50] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.17, 15.58, 13.02 [15:34:47] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.23, 14.72, 13.02 [15:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerPC [15:35:09] [02miraheze/services] 07MirahezeSSLBot 03a2fc93f - BOT: Updating services config for wikis [15:36:44] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 10.26, 13.50, 12.80 [15:50:31] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.30, 13.66, 13.18 [15:52:27] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 9.00, 11.97, 12.61 [16:05:32] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:35] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2246 MB (9% inode=94%); [16:24:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [16:26:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:36:19] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.25, 14.59, 13.04 [16:37:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JerXa [16:37:10] [02miraheze/mw-config] 07paladox 0343b48f0 - Move all wikis that begin with the letter c to the new mount [16:37:12] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:37:13] [02mw-config] 07paladox opened pull request 03#2800: Move all wikis that begin with the letter c to the new mount - 13https://git.io/JerXV [16:38:16] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 5.99, 12.13, 12.40 [16:39:29] [02mw-config] 07paladox closed pull request 03#2800: Move all wikis that begin with the letter c to the new mount - 13https://git.io/JerXV [16:39:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerXi [16:39:32] [02miraheze/mw-config] 07paladox 0381287a7 - Move all wikis that begin with the letter c to the new mount (#2800) [16:40:33] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:40:34] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:53:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.94, 14.92, 13.30 [16:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jer1n [16:55:08] [02miraheze/services] 07MirahezeSSLBot 03ffe4cb9 - BOT: Updating services config for wikis [16:57:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.18, 15.13, 13.71 [16:59:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 5.80, 11.58, 12.59 [17:06:38] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static-new] [17:10:58] paladox: ^ [17:11:05] yup [17:12:40] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:13:34] That’s better [17:17:46] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf a* [17:17:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:23:27] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf b* [17:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:25:18] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf c* [17:25:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:35:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.47, 13.57, 9.72 [17:37:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.34, 16.63, 11.31 [17:47:47] * hispano76 saluda [18:03:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.71, 15.05, 15.95 [18:06:35] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [18:09:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 22.89, 17.88, 16.76 [18:23:24] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:29:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JerDh [18:29:23] [02miraheze/mw-config] 07paladox 03aa72c1e - Move all wikis that start with the letter d to the new mount [18:29:25] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:29:26] [02mw-config] 07paladox opened pull request 03#2801: Move all wikis that start with the letter d to the new mount - 13https://git.io/JerDj [18:29:39] [02mw-config] 07paladox closed pull request 03#2801: Move all wikis that start with the letter d to the new mount - 13https://git.io/JerDj [18:29:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jerye [18:29:42] [02miraheze/mw-config] 07paladox 030db905f - Move all wikis that start with the letter d to the new mount (#2801) [18:30:14] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [18:30:16] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [18:31:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 8.55, 11.68, 15.47 [18:35:11] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.72, 14.77, 15.86 [18:39:11] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.83, 14.98, 15.71 [18:41:14] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 35.43, 20.77, 17.63 [18:51:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 6.17, 12.49, 15.42 [18:55:06] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 5.24, 9.08, 13.38 [18:55:40] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx],Service[nginx-prometheus-exporter] [18:56:13] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf d* [18:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:04:02] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:05:41] .task 1 [19:05:43] https://phabricator.miraheze.org/T1 - Tracking: Deployment of Miraheze [Resolved] authored by John, assigned to github-migration [19:06:20] .task T4882 RhinosF1 [19:06:21] hispano76: I can't find task with id 48821 [19:06:28] .task 4882 RhinosF1 [19:06:28] hispano76: I can't find task with id 48821 [19:10:42] RhinosF1: see Phab task 4882 [19:53:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.41, 12.50, 8.64 [19:56:26] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx],Service[nginx-prometheus-exporter] [20:02:38] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:03:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 9.12, 14.81, 12.73 [20:05:27] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 26.74, 17.82, 14.00 [20:15:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.55, 15.84, 15.09 [20:19:17] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 15.49, 16.26, 15.44 [20:21:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.66, 15.39, 15.24 [20:23:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.00, 17.81, 16.16 [20:25:10] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static-new] [20:31:32] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 7.39, 13.73, 15.39 [20:33:06] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:37:37] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 5.56, 8.44, 12.47 [20:44:15] hispano76: if it does it gain give us a shout [20:44:42] *again [20:45:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jer9R [20:46:00] [02miraheze/mw-config] 07paladox 032993d13 - Move all wikis that begin with the letter e to the new mount [20:46:02] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:46:03] [02mw-config] 07paladox opened pull request 03#2802: Move all wikis that begin with the letter e to the new mount - 13https://git.io/Jer90 [20:46:08] [02mw-config] 07paladox closed pull request 03#2802: Move all wikis that begin with the letter e to the new mount - 13https://git.io/Jer90 [20:46:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jer9E [20:46:11] [02miraheze/mw-config] 07paladox 035ad5b5a - Move all wikis that begin with the letter e to the new mount (#2802) [20:52:43] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.95, 15.15, 12.40 [20:54:42] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.78, 14.58, 12.56 [20:58:37] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 9.46, 12.76, 12.32 [21:10:44] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 22.35, 18.45, 14.99 [21:12:41] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 7.88, 14.37, 13.91 [21:14:39] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 5.19, 11.24, 12.82 [21:34:59] just wondering, whats up with this dashboard name [21:34:59] https://grafana.miraheze.org/d/72sSwXpWk/php-fpm66666666?orgId=1 [21:35:01] [ Grafana ] - grafana.miraheze.org [21:41:57] apoop i'm going to delete that dashboard. [21:41:59] was a test [21:49:56] oh ok [21:55:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JerHS [21:55:07] [02miraheze/mw-config] 07paladox 03019a092 - Move all wikis that start with the letter f to the new mount [21:55:08] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [21:55:10] [02mw-config] 07paladox opened pull request 03#2803: Move all wikis that start with the letter f to the new mount - 13https://git.io/JerH9 [21:55:47] [02mw-config] 07paladox synchronize pull request 03#2803: Move all wikis that start with the letter f to the new mount - 13https://git.io/JerH9 [21:55:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JerHH [21:55:50] [02miraheze/mw-config] 07paladox 031b5be30 - Update LocalSettings.php [21:57:00] miraheze/mw-config/paladox-patch-2/1b5be30 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/612091937 [21:58:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.99, 11.20, 7.91 [22:02:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.95, 13.89, 9.93 [22:04:12] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.77, 16.88, 11.58 [22:20:23] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 6.82, 14.79, 14.69 [22:21:52] [02mw-config] 07paladox closed pull request 03#2803: Move all wikis that start with the letter f to the new mount - 13https://git.io/JerH9 [22:21:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerQn [22:21:55] [02miraheze/mw-config] 07paladox 038ed5fd3 - Move all wikis that start with the letter f to the new mount (#2803) * Move all wikis that start with the letter f to the new mount * Update LocalSettings.php [22:22:07] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:22:18] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:22:19] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [22:22:24] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 5.21, 11.47, 13.48 [22:22:31] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [22:24:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:26:27] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.394 second response time [22:27:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.37, 15.91, 14.91 [22:29:23] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.12, 15.90, 15.11 [22:30:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [22:30:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [22:31:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.65, 16.84, 15.52 [22:32:10] What is the new support? "new mount " [22:35:25] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.74, 15.94, 15.59 [22:36:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:36:56] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:39:18] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.45, 9.76, 13.14 [22:42:08] https://news.sky.com/story/general-election-labour-promises-free-broadband-if-they-win-election-11861545 [22:42:09] [ General election: Labour promises free broadband if party wins election | Politics News | Sky News ] - news.sky.com [22:43:51] oh [22:43:53] wrong channel [22:55:42] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf e* [22:55:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:56:22] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf f* [22:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:57:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jer70 [22:57:38] [02miraheze/mw-config] 07paladox 037ed5ef9 - Move all wikis that start with the letter g to the new mount [22:57:39] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:57:41] [02mw-config] 07paladox opened pull request 03#2804: Move all wikis that start with the letter g to the new mount - 13https://git.io/Jer7E [22:58:00] hispano76 it's a new file storage system we are rolling out, replaces lizardfs with gluster. [23:01:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.87, 13.96, 10.93 [23:03:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.99, 13.89, 11.26 [23:07:10] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.89, 15.91, 12.67