[00:03:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.62, 15.30, 15.90 [00:13:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.03, 13.68, 14.37 [00:17:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 9.60, 12.36, 13.73 [00:21:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.70, 14.48, 14.32 [00:25:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.05, 14.41, 14.40 [00:45:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 9.77, 12.41, 13.31 [01:30:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerFO [01:30:38] [02miraheze/mw-config] 07paladox 03fac4ef7 - Update LocalSettings.php [01:44:28] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.01, 13.55, 12.40 [01:46:26] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.45, 14.07, 12.71 [01:50:19] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 13.31, 13.37, 12.71 [02:25:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.48, 15.55, 13.66 [02:27:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.36, 15.64, 13.96 [02:29:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.91, 16.22, 14.35 [02:31:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.61, 15.47, 14.29 [02:39:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:10] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.26, 15.91, 15.26 [02:55:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.89, 14.14, 14.18 [02:57:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.14, 14.07, 14.19 [03:03:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.55, 14.40, 14.23 [03:05:09] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.76, 13.55, 13.96 [03:11:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 11.06, 12.88, 13.60 [03:19:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.53, 14.40, 14.00 [03:21:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.45, 15.15, 14.31 [03:29:11] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.27, 15.40, 14.85 [03:31:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.13, 17.04, 15.49 [03:35:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 10.05, 15.47, 15.34 [03:39:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 2.49, 8.59, 12.56 [05:40:06] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JerNd [05:40:08] [02miraheze/services] 07MirahezeSSLBot 03efd4753 - BOT: Updating services config for wikis [06:26:58] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2656 MB (11% inode=94%); [06:37:05] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [09:20:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jerhf [09:20:09] [02miraheze/services] 07MirahezeSSLBot 036b2089a - BOT: Updating services config for wikis [11:23:34] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 2 minutes ago with 13 failures. Failed resources (up to 3 shown) [11:23:40] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:23:41] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:23:59] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 2 minutes ago with 20 failures. Failed resources (up to 3 shown) [11:24:06] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 2 minutes ago with 17 failures. Failed resources (up to 3 shown) [11:24:09] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 212 failures. Last run 2 minutes ago with 212 failures. Failed resources (up to 3 shown) [11:24:27] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 2 minutes ago with 21 failures. Failed resources (up to 3 shown) [11:24:41] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 233 failures. Last run 2 minutes ago with 233 failures. Failed resources (up to 3 shown) [11:24:41] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [11:24:42] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 229 failures. Last run 2 minutes ago with 229 failures. Failed resources (up to 3 shown) [11:24:51] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:24:58] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [11:25:05] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 213 failures. Last run 3 minutes ago with 213 failures. Failed resources (up to 3 shown) [11:25:09] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 227 failures. Last run 3 minutes ago with 227 failures. Failed resources (up to 3 shown) [11:25:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown) [11:25:19] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 3 minutes ago with 32 failures. Failed resources (up to 3 shown) [11:25:41] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 3 minutes ago with 51 failures. Failed resources (up to 3 shown) [11:25:44] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 413 failures. Last run 3 minutes ago with 413 failures. Failed resources (up to 3 shown) [11:33:55] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [11:34:03] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:34:24] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [11:34:36] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [11:34:50] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [11:34:56] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:07] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [11:35:12] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:24] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:34] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:41] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:42] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [11:35:42] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:35:49] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:36:07] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:36:36] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:36:58] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [11:37:08] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:40:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoeX [11:40:10] [02miraheze/services] 07MirahezeSSLBot 03711abd0 - BOT: Updating services config for wikis [12:58:11] [02mw-config] 07paladox closed pull request 03#2804: Move all wikis that start with the letter g to the new mount - 13https://git.io/Jer7E [12:58:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeovS [12:58:14] [02miraheze/mw-config] 07paladox 03f95f7b9 - Move all wikis that start with the letter g to the new mount (#2804) [13:09:59] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [13:10:00] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [13:10:34] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf g* [13:10:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:15:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.48, 12.20, 7.65 [13:17:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.62, 13.48, 8.64 [13:25:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.88, 15.94, 12.22 [13:27:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.59, 17.25, 13.20 [13:29:09] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.67, 15.98, 13.23 [13:37:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.08, 16.75, 14.30 [13:39:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeofK [13:39:24] [02miraheze/mw-config] 07paladox 03e114c78 - Move all wikis that start with the letter h to the new mount [13:41:06] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.12, 15.54, 14.35 [13:55:17] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.55, 16.04, 14.71 [13:57:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.91, 15.58, 14.73 [14:05:18] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.19, 15.86, 15.11 [14:07:18] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.46, 15.54, 15.09 [14:11:16] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.36, 9.45, 12.70 [14:11:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoJ8 [14:11:26] [02miraheze/mw-config] 07paladox 03daf0003 - Move all wikis that start with the letter i to the new mount [14:11:28] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [14:11:29] [02mw-config] 07paladox opened pull request 03#2805: Move all wikis that start with the letter i to the new mount - 13https://git.io/JeoJ4 [14:11:53] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf h* [14:12:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:15:18] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.26, 14.66, 14.00 [14:45:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 5.08, 12.52, 15.97 [14:49:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.25, 8.39, 13.52 [14:49:58] [02mw-config] 07paladox closed pull request 03#2805: Move all wikis that start with the letter i to the new mount - 13https://git.io/JeoJ4 [14:50:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoJh [14:50:01] [02miraheze/mw-config] 07paladox 03f05b906 - Move all wikis that start with the letter i to the new mount (#2805) [14:50:38] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [14:50:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [14:55:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoUq [14:55:09] [02miraheze/services] 07MirahezeSSLBot 03fc1db94 - BOT: Updating services config for wikis [15:00:48] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.20, 15.22, 14.39 [15:04:48] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.57, 15.94, 14.86 [15:08:41] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.99, 15.25, 14.81 [15:14:38] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 18.61, 15.96, 15.22 [15:18:32] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 8.64, 13.45, 14.48 [15:20:35] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 6.04, 10.83, 13.39 [15:22:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoUM [15:22:53] [02miraheze/mw-config] 07paladox 03ea503b6 - Move all wikis that start with the letter j to the new mount [15:23:24] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf i* [15:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:23:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoUy [15:23:52] [02miraheze/mw-config] 07paladox 0305e9e79 - Revert "Move all wikis that start with the letter j to the new mount" This reverts commit ea503b68442a30d26a608e0a7bf8f5f36f22cff6. [15:24:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoUS [15:24:24] [02miraheze/mw-config] 07paladox 03533fe2f - Move all wikis that start with the letter j to the new mount [15:24:26] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [15:24:27] [02mw-config] 07paladox opened pull request 03#2806: Move all wikis that start with the letter j to the new mount - 13https://git.io/JeoU9 [15:28:54] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.55, 16.62, 14.59 [15:37:03] I am currently testing twinkle on meta - if something breaks, let me know [15:45:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 10.58, 14.52, 15.92 [15:45:10] [02mw-config] 07paladox closed pull request 03#2806: Move all wikis that start with the letter j to the new mount - 13https://git.io/JeoU9 [15:45:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoTB [15:45:13] [02miraheze/mw-config] 07paladox 035fbbdd5 - Move all wikis that start with the letter j to the new mount (#2806) [15:53:14] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:56:54] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [15:56:56] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [15:57:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 8.12, 14.69, 15.63 [15:57:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoTy [15:57:21] [02miraheze/mw-config] 07paladox 03e1f704d - Move all wikis that start with the letter k to the new mount [15:57:23] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [15:57:46] [02mw-config] 07paladox opened pull request 03#2807: Move all wikis that start with the letter k to the new mount - 13https://git.io/JeoTS [15:58:15] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf j* [15:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:01:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.19, 14.90, 15.34 [16:05:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.89, 15.52, 15.56 [16:13:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 23.38, 17.11, 15.87 [16:15:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.74, 15.94, 15.56 [16:17:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.97, 18.02, 16.39 [16:55:06] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeokh [16:55:08] [02miraheze/services] 07MirahezeSSLBot 03dee290c - BOT: Updating services config for wikis [17:03:04] [02mw-config] 07paladox closed pull request 03#2807: Move all wikis that start with the letter k to the new mount - 13https://git.io/JeoTS [17:03:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoIY [17:03:07] [02miraheze/mw-config] 07paladox 03c071394 - Move all wikis that start with the letter k to the new mount (#2807) [17:03:18] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 4.80, 11.70, 15.60 [17:03:20] paladox: Reception123 RhinosF1 Zppix https://phabricator.miraheze.org/T4903#93184 You can relieve me, dont seem to understood what Im saying. [17:03:22] [ ⚓ T4903 Free images ] - phabricator.miraheze.org [17:05:49] ok [17:07:16] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 6.22, 8.16, 13.23 [17:16:37] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf k* [17:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:19:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.92, 11.90, 11.28 [17:19:43] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [17:19:44] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [17:20:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoIV [17:20:13] [02miraheze/mw-config] 07paladox 03e91f404 - Move all wikis that start with the letter l to the new mount [17:20:15] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [17:20:16] [02mw-config] 07paladox opened pull request 03#2808: Move all wikis that start with the letter l to the new mount - 13https://git.io/JeoIw [17:39:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoIh [17:39:56] [02miraheze/mw-config] 07paladox 03a8e3845 - Update LocalWiki.php [17:43:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoLT [17:43:21] [02miraheze/mw-config] 07paladox 03927cdcb - Enable $wgGenerateThumbnailOnParse for rhinosf1wiki [17:53:18] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:19] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:55:16] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.236 second response time [17:55:19] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 5.068 second response time [18:07:12] !log restart php7.2-fpm on mw2 [18:07:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:08:21] !log depool mw2 && reboot mw2 && repool [18:08:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:14:27] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 6.20, 11.13, 15.04 [18:15:17] [02mw-config] 07paladox closed pull request 03#2808: Move all wikis that start with the letter l to the new mount - 13https://git.io/JeoIw [18:15:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoLD [18:15:20] [02miraheze/mw-config] 07paladox 039d982e5 - Move all wikis that start with the letter l to the new mount (#2808) [18:16:11] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:16:13] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [18:16:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoLH [18:16:50] [02miraheze/mw-config] 07paladox 03ba2d3e2 - Move all wikis that start with the letter n to the new mount [18:16:51] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:16:53] [02mw-config] 07paladox opened pull request 03#2809: Move all wikis that start with the letter n to the new mount - 13https://git.io/JeoLQ [18:17:29] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf l* [18:17:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:18:23] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 20.55, 12.02, 14.25 [18:20:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.07, 12.69, 14.26 [18:22:19] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.83, 14.85, 14.88 [18:29:49] paladox: did you ever sort out access to the Mediawiki stuff on LFS6 for mw-admin [18:30:08] no, as i said need all ops to approve [18:30:15] I'll phab it [18:32:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:32:50] paladox: https://phabricator.miraheze.org/T4904#93194 [18:32:51] [ ⚓ T4904 Allow mw-admins access to Mediawiki areas of Lizardfs6 ] - phabricator.miraheze.org [18:33:04] ok [18:36:15] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:38:21] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 4.521 second response time [18:38:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:02:18] Apologies, MirahezeRC flooding is my fault. [19:16:42] !log sudo -u www-data php /srv/mediawiki/w/maintenance/updateSpecialPages.php --override --wiki=metawiki [19:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:17:18] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=metawiki (import and some stat issue with translations [19:17:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:27:57] .mh Code_Of_Conduct/ja [19:27:57] https://meta.miraheze.org/wiki/Code_Of_Conduct/ja [19:45:13] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [19:48:01] ^ see -staff [19:53:16] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:56:23] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-1 [+0/-0/±1] 13https://git.io/JeomG [19:56:25] [02miraheze/mediawiki] 07RhinosF1 03c902068 - Updates [19:56:26] [02mediawiki] 07RhinosF1 created branch 03RhinosF1-1 - 13https://git.io/vbL5b [19:56:53] [02mediawiki] 07RhinosF1 deleted branch 03RhinosF1-1 - 13https://git.io/vbL5b [19:56:54] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03RhinosF1-1 [19:57:03] nothing on it [20:00:44] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:07] !log rhinos@mw1:~$ sudo -u www-data nice -5 php /srv/mediawiki/w/maintenance/refreshLinks.php 17600 --wiki=metawiki [20:01:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:03:13] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:34] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:15] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.558 second response time [20:07:32] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.003 second response time [20:11:28] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 1.393 second response time [20:12:31] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [20:14:27] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.04, 14.78, 15.74 [20:15:13] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:35] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:27] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:17:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:17:16] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.398 second response time [20:17:28] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.004 second response time [20:18:26] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.399 second response time [20:20:52] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 22.00, 16.24, 15.86 [20:22:52] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.01, 15.31, 15.54 [20:24:54] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.04, 16.39, 15.89 [20:36:40] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.39, 15.51, 15.86 [20:40:36] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.53, 16.38, 16.11 [21:13:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [21:15:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:23:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [21:25:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:37:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:39:13] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:41:13] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:43:23] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.752 second response time [21:43:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:45:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:45:28] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 7.07, 10.58, 14.94 [21:45:54] [02mw-config] 07paladox closed pull request 03#2809: Move all wikis that start with the letter n to the new mount - 13https://git.io/JeoLQ [21:45:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoOP [21:45:57] [02miraheze/mw-config] 07paladox 03c4501c7 - Move all wikis that start with the letter n to the new mount (#2809) [21:45:59] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [21:46:00] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [21:49:23] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 6.55, 8.35, 13.02 [21:58:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoOb [21:58:13] [02miraheze/mw-config] 07paladox 03f76db51 - Move all wikis that start with the letter m to the new mount [21:58:14] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [21:58:16] [02mw-config] 07paladox opened pull request 03#2810: Move all wikis that start with the letter m to the new mount - 13https://git.io/JeoON [21:58:50] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf n* [21:58:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:03:07] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 24.20, 13.15, 11.36 [22:09:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.09, 14.62, 12.86 [22:10:42] paladox: Why does creating a wiki take so much longer than declining one? [22:11:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.10, 15.01, 13.19 [22:11:21] Well declining is just a simple mysql update to the table to indicate it, where as create it has to run various things like scripts [22:12:10] ah that makes sense [22:15:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.55, 15.51, 13.89 [22:17:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.22, 17.51, 14.80 [22:26:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo36 [22:26:29] [02miraheze/puppet] 07paladox 03e2a5327 - Update mfsmaster.cfg.erb [22:27:35] !log restart lizardfs-master on lizardfs6 [22:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:29:13] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 5.49, 13.21, 14.88 [22:29:20] !log restart lizardfs-chunkserver on lizardfs* [22:29:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:30:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo3y [22:30:10] [02miraheze/services] 07MirahezeSSLBot 03865c9bf - BOT: Updating services config for wikis [22:31:17] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 3.87, 9.81, 13.41 [22:39:09] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.08, 13.35, 13.22 [22:41:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.40, 14.14, 13.62 [22:43:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.05, 15.61, 14.23 [22:45:05] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.97, 15.77, 14.49 [22:47:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.91, 17.79, 15.41