[00:13:40] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.69, 1.74, 1.11 [00:15:40] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.55, 1.29, 1.02 [00:17:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.14, 3.72, 3.98 [00:23:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.10, 3.72, 3.87 [00:33:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.60, 3.92, 3.97 [00:35:43] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqC8U [00:35:45] [02miraheze/ManageWiki] 07Universal-Omega 0385fcb36 - use onContentHandlerForModelID hook to handle unknown content models [00:35:46] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [00:35:48] [02ManageWiki] 07Universal-Omega opened pull request 03#257: use onContentHandlerForModelID hook to handle unknown content models - 13https://git.io/JqC8I [00:36:46] miraheze/ManageWiki - Universal-Omega the build passed. [00:37:08] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqC8q [00:37:09] [02miraheze/ManageWiki] 07Universal-Omega 03f162a36 - Update extension.json [00:37:11] [02ManageWiki] 07Universal-Omega synchronize pull request 03#257: use onContentHandlerForModelID hook to handle unknown content models - 13https://git.io/JqC8I [00:38:11] miraheze/ManageWiki - Universal-Omega the build passed. [00:39:17] [02ManageWiki] 07Universal-Omega closed pull request 03#257: use onContentHandlerForModelID hook to handle unknown content models - 13https://git.io/JqC8I [00:39:19] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqC8s [00:39:20] [02miraheze/ManageWiki] 07Universal-Omega 03a58bf93 - use onContentHandlerForModelID hook to handle unknown content models (#257) [00:39:22] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:39:23] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [00:39:57] [02ManageWiki] 07Universal-Omega closed pull request 03#256: Remove maintainprefix stuff - 13https://git.io/JqCWc [00:40:00] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:40:01] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [00:40:16] miraheze/ManageWiki - Universal-Omega the build passed. [00:41:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.15, 3.87, 3.89 [00:41:45] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqC8W [00:41:47] [02miraheze/ManageWiki] 07Universal-Omega 033818036 - Remove contentmodel stuff from NamespaceMigrationJob [00:41:48] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [00:41:50] [02ManageWiki] 07Universal-Omega opened pull request 03#258: Remove contentmodel stuff from NamespaceMigrationJob - 13https://git.io/JqC8l [00:42:50] miraheze/ManageWiki - Universal-Omega the build passed. [00:43:57] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqC8z [00:43:59] [02miraheze/ManageWiki] 07Universal-Omega 03c2d70f1 - Update NamespaceMigrationJob.php [00:44:00] [02ManageWiki] 07Universal-Omega synchronize pull request 03#258: Remove contentmodel stuff from NamespaceMigrationJob - 13https://git.io/JqC8l [00:44:56] miraheze/ManageWiki - Universal-Omega the build passed. [00:45:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.31, 3.81, 3.88 [00:45:29] [02ManageWiki] 07Universal-Omega closed pull request 03#258: Remove contentmodel stuff from NamespaceMigrationJob - 13https://git.io/JqC8l [00:45:30] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqC8o [00:45:31] [02miraheze/ManageWiki] 07Universal-Omega 03b75cd90 - Remove contentmodel stuff from NamespaceMigrationJob (#258) [00:45:33] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [00:45:34] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:46:26] miraheze/ManageWiki - Universal-Omega the build passed. [00:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.82, 4.04, 3.92 [00:56:21] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqC8x [00:56:23] [02miraheze/ManageWiki] 07Universal-Omega 03b859693 - use TextContentHandler as fallback rather than WikitextContentHandler [00:57:20] miraheze/ManageWiki - Universal-Omega the build passed. [00:59:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqC4t [00:59:54] [02miraheze/mediawiki] 07Universal-Omega 0346354f4 - Update ManageWiki [00:59:55] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:00:55] [02mediawiki] 07Universal-Omega opened pull request 03#1279: Update ManageWiki - 13https://git.io/JqC4q [01:04:12] [02mediawiki] 07Universal-Omega closed pull request 03#1279: Update ManageWiki - 13https://git.io/JqC4q [01:04:14] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqC43 [01:04:15] [02miraheze/mediawiki] 07Universal-Omega 0359239fc - Update ManageWiki (#1279) [01:04:17] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:04:18] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:04:23] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 191s [01:06:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 283s [01:16:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 40s [01:22:23] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 223s [01:24:32] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCBe [01:24:34] [02miraheze/WikiDiscover] 07Universal-Omega 037a0ac8e - fix wiki_deleted and wiki_locked: Undefined property: stdClass [01:25:26] miraheze/WikiDiscover - Universal-Omega the build passed. [01:27:42] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqCBq [01:27:44] [02miraheze/mediawiki] 07Universal-Omega 032eebf27 - Update WikiDiscover [01:41:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.34, 3.78, 3.95 [01:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.08, 3.94, 3.95 [01:51:36] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.22, 3.56, 2.91 [01:53:34] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.90, 3.26, 2.88 [02:17:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.91, 3.56, 3.96 [02:22:21] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.48, 3.54, 3.15 [02:23:48] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCR6 [02:23:49] [02miraheze/ManageWiki] 07Universal-Omega 039557f3f - Add 'settings' to ManageWikiRequirements [02:24:19] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.70, 3.29, 3.11 [02:24:47] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCR1 [02:24:49] [02miraheze/WikiDiscover] 07Universal-Omega 03b0051c5 - Remove unused function [02:24:51] miraheze/ManageWiki - Universal-Omega the build passed. [02:25:41] miraheze/WikiDiscover - Universal-Omega the build passed. [02:29:08] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.43, 4.10, 3.92 [02:29:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqCR9 [02:29:58] [02miraheze/mediawiki] 07Universal-Omega 031c10454 - Update ManageWiki [02:31:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqCR7 [02:31:38] [02miraheze/mediawiki] 07Universal-Omega 0396e803d - Update WikiDiscover [03:02:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqC0z [03:02:06] [02miraheze/mw-config] 07Universal-Omega 03f449d07 - Add validation for shared uploads [03:02:07] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [03:02:09] [02mw-config] 07Universal-Omega opened pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [03:03:11] miraheze/mw-config - Universal-Omega the build passed. [03:18:23] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 113s [03:18:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqC0A [03:18:39] [02miraheze/mw-config] 07Universal-Omega 03f173b64 - Update ManageWikiSettings.php [03:18:41] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [03:19:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqC0x [03:19:41] miraheze/mw-config - Universal-Omega the build passed. [03:19:42] [02miraheze/mw-config] 07Universal-Omega 03c1bb828 - Update LocalSettings.php [03:19:44] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [03:20:45] miraheze/mw-config - Universal-Omega the build passed. [03:22:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [04:38:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 204s [04:39:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.28, 3.53, 3.92 [04:40:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [04:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.07, 3.91, 3.88 [05:04:50] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 5.78, 4.66, 3.18 [05:12:50] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 5.03, 5.06, 4.04 [05:27:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.26, 3.57, 3.97 [05:29:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.18, 4.06, 4.09 [05:37:19] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 6.81, 3.76, 1.68 [05:39:19] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.19, 3.08, 1.68 [05:54:28] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCz9 [05:54:30] [02miraheze/WikiDiscover] 07Universal-Omega 03186a5ef - Delete sonar-project.properties [05:55:26] miraheze/WikiDiscover - Universal-Omega the build passed. [05:55:37] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCz7 [05:55:38] [02miraheze/ManageWiki] 07Universal-Omega 03acbfbfd - Delete sonar-project.properties [05:56:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCz5 [05:56:19] [02miraheze/CreateWiki] 07Universal-Omega 0328483c0 - Delete sonar-project.properties [05:56:30] miraheze/ManageWiki - Universal-Omega the build passed. [05:57:11] miraheze/CreateWiki - Universal-Omega the build passed. [05:58:00] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCzd [05:58:02] [02miraheze/DataDump] 07Universal-Omega 03cc6c400 - Delete sonar-project.properties [05:58:52] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCzF [05:58:54] [02miraheze/MatomoAnalytics] 07Universal-Omega 0305944b7 - Delete sonar-project.properties [05:58:55] miraheze/DataDump - Universal-Omega the build passed. [05:59:11] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCzb [05:59:13] [02miraheze/RottenLinks] 07Universal-Omega 03511fdfd - Delete sonar-project.properties [05:59:32] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JqCzA [05:59:33] [02miraheze/IncidentReporting] 07Universal-Omega 03e35d9c8 - Delete sonar-project.properties [05:59:45] miraheze/MatomoAnalytics - Universal-Omega the build passed. [06:00:06] miraheze/RottenLinks - Universal-Omega the build passed. [06:00:26] miraheze/IncidentReporting - Universal-Omega the build passed. [06:04:34] RECOVERY - phab2 APT on phab2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:05:02] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCzh [06:05:03] [02miraheze/GlobalNewFiles] 07Universal-Omega 03d831a1a - Use RemoteWiki instead of GlobalVarConfig [06:05:56] miraheze/GlobalNewFiles - Universal-Omega the build passed. [06:05:57] RECOVERY - cp12 APT on cp12 is OK: APT OK: 20 packages available for upgrade (0 critical updates). [06:11:22] RECOVERY - db13 APT on db13 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [06:11:55] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCgT [06:11:56] [02miraheze/CreateWiki] 07Universal-Omega 03d016cf9 - Add license-name [06:12:50] miraheze/CreateWiki - Universal-Omega the build passed. [06:13:08] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCgL [06:13:09] [02miraheze/ManageWiki] 07Universal-Omega 03ce4f5ee - Add license-name [06:13:13] RECOVERY - cp11 APT on cp11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:13:46] RECOVERY - ns2 APT on ns2 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [06:14:01] miraheze/ManageWiki - Universal-Omega the build passed. [06:14:28] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCgt [06:14:30] [02miraheze/IncidentReporting] 07Universal-Omega 036cb6ba0 - Add license-name [06:15:26] miraheze/IncidentReporting - Universal-Omega the build passed. [06:16:12] RECOVERY - services4 APT on services4 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [06:16:50] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCgO [06:16:51] [02miraheze/IncidentReporting] 07Universal-Omega 03a22eb95 - Bump version to 1.1.4 [06:16:53] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [06:16:54] [02IncidentReporting] 07Universal-Omega opened pull request 03#16: Bump version to 1.1.4 - 13https://git.io/JqCg3 [06:17:56] miraheze/IncidentReporting - Universal-Omega the build passed. [06:17:57] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCgn [06:17:59] [02miraheze/IncidentReporting] 07Universal-Omega 037ceb762 - Update extension.json [06:18:00] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#16: Bump version to 1.1.4 - 13https://git.io/JqCg3 [06:18:28] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqCgc [06:18:30] [02miraheze/IncidentReporting] 07Universal-Omega 03633ceae - Bump version to 1.1.4 (#16) [06:18:31] [02IncidentReporting] 07Universal-Omega closed pull request 03#16: Bump version to 1.1.4 - 13https://git.io/JqCg3 [06:18:33] [02IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [06:18:34] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:19:06] miraheze/IncidentReporting - Universal-Omega the build passed. [06:19:26] miraheze/IncidentReporting - Universal-Omega the build passed. [06:19:32] RECOVERY - cp3 APT on cp3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:19:56] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCgC [06:19:57] [02miraheze/MatomoAnalytics] 07Universal-Omega 037b54ae2 - Add license-name [06:19:59] [02MatomoAnalytics] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fN4LT [06:20:01] [02MatomoAnalytics] 07Universal-Omega opened pull request 03#41: Add license-name - 13https://git.io/JqCgW [06:20:56] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCgl [06:20:57] [02miraheze/MatomoAnalytics] 07Universal-Omega 03635ac90 - Update CHANGELOG [06:20:59] [02MatomoAnalytics] 07Universal-Omega synchronize pull request 03#41: Add license-name - 13https://git.io/JqCgW [06:21:00] miraheze/MatomoAnalytics - Universal-Omega the build passed. [06:21:50] miraheze/MatomoAnalytics - Universal-Omega the build passed. [06:21:58] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#41: Add license-name - 13https://git.io/JqCgW [06:21:59] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqCgB [06:22:00] [02miraheze/MatomoAnalytics] 07Universal-Omega 03ec3deed - Add license-name (#41) [06:22:02] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:22:04] [02MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fN4LT [06:22:37] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:22:55] miraheze/MatomoAnalytics - Universal-Omega the build passed. [06:23:28] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JqCgR [06:23:29] [02miraheze/WikiDiscover] 07Universal-Omega 0377fabd6 - Rename COPYING to LICENSE [06:24:00] RECOVERY - db11 APT on db11 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [06:24:04] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [06:24:11] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:24:26] miraheze/WikiDiscover - Universal-Omega the build passed. [06:25:31] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [06:26:04] [02miraheze/CustomHeader] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCg2 [06:26:05] [02miraheze/CustomHeader] 07Universal-Omega 0348e0235 - Add license-name [06:27:00] miraheze/CustomHeader - Universal-Omega the build passed. [06:28:02] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:28:55] RECOVERY - ns1 APT on ns1 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [06:34:14] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:35:02] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:35:02] RECOVERY - services3 APT on services3 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [06:37:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.76, 3.84, 3.99 [06:37:09] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:39:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.47, 4.04, 4.05 [06:40:03] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:45:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.83, 3.91, 4.00 [06:45:27] RECOVERY - mw11 APT on mw11 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:46:04] RECOVERY - mw10 APT on mw10 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:49:22] RECOVERY - mon2 APT on mon2 is OK: APT OK: 6 packages available for upgrade (0 critical updates). [06:49:48] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [06:49:57] RECOVERY - db12 APT on db12 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [06:50:43] miraheze/mw-config - R4356th the build passed. [06:51:39] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [06:52:09] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 20 packages available for upgrade (0 critical updates). [06:53:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.57, 4.08, 4.01 [06:55:03] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 20 packages available for upgrade (0 critical updates). [06:58:19] RECOVERY - mem2 APT on mem2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:58:20] RECOVERY - mem1 APT on mem1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:58:23] RECOVERY - cp10 APT on cp10 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [06:58:48] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [06:59:45] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 96 packages available for upgrade (0 critical updates). [07:00:19] RECOVERY - mail2 APT on mail2 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [07:01:40] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [07:14:10] [02mw-config] 07R4356th synchronize pull request 03#3761: Better canned responses - 13https://git.io/JqIWl [07:15:09] miraheze/mw-config - R4356th the build passed. [07:19:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.16, 3.56, 3.89 [07:23:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.44, 3.93, 3.96 [07:23:51] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqC2F [07:23:52] [02miraheze/WikiDiscover] 07Universal-Omega 03a258459 - Fix license-name [07:24:46] miraheze/WikiDiscover - Universal-Omega the build passed. [07:26:41] [02miraheze/DataDump] 07Universal-Omega deleted branch 03paladox-patch-1 [07:26:43] [02DataDump] 07Universal-Omega deleted branch 03paladox-patch-1 - 13https://git.io/fhhKV [07:29:40] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:26] [02mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [07:30:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03RhinosF1-patch-1 [07:30:29] [02mw-config] 07Universal-Omega deleted branch 03revert-3756-Universal-Omega-patch-3 - 13https://git.io/vbvb3 [07:30:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-3756-Universal-Omega-patch-3 [07:33:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.61, 3.93, 3.99 [07:42:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 179s [07:43:11] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.16, 3.45, 2.31 [07:45:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.50, 3.81, 3.79 [07:45:12] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.88, 3.27, 2.39 [07:51:12] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.42, 3.54, 2.80 [07:52:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [07:53:14] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.02, 2.56, 2.53 [07:57:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.25, 3.74, 3.87 [07:59:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.52, 4.12, 4.00 [07:59:13] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.41, 2.77, 2.56 [08:01:12] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.80, 3.06, 2.71 [08:19:11] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.74, 3.83, 1.79 [08:21:06] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.87, 2.70, 1.62 [08:23:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.29, 3.58, 3.94 [08:29:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.28, 3.74, 3.86 [09:13:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.45, 3.63, 4.00 [09:17:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.13, 3.76, 3.95 [09:21:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.55, 3.81, 3.95 [09:31:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.36, 3.95, 3.92 [09:33:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.51, 3.78, 3.86 [09:37:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.48, 4.11, 3.96 [10:30:17] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=trollpastawiki --r "[[phab:T6950|Requested]]" /home/reception/trollpastadel.txt [10:30:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.38, 3.61, 3.97 [10:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.01, 4.00, 3.98 [11:32:52] Reception123: how much is in that delete page script? [11:37:22] RhinosF1: a lot... 10k+ [11:37:53] Reception123: it's messed up #miraheze-feed [11:37:59] Is there a delay parameter [11:38:21] I think there would be an interval one [11:38:29] maybe we can try like 3 seconds in between or something like that [11:39:15] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 5.14, 3.43, 2.02 [11:40:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 191s [11:41:16] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.46, 3.47, 2.22 [11:42:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 258s [11:42:25] Reception123: good idea [11:44:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 43s [11:45:18] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.86, 3.28, 2.46 [11:46:16] RhinosF1: doesn't seem to work though for some reason, not sure what other format they'd want other than "3s" or "3" [11:46:25] it doesn't seem to be reacting to it [11:46:47] and --help doesn't give an example of the format [12:25:44] @R4356th: https://phabricator.miraheze.org/T6956#137433 [12:25:46] [ Login ] - phabricator.miraheze.org [12:31:30] Ack. [12:31:44] Also, please do read the description. [12:32:02] @R4356th: https://phabricator.miraheze.org/T6956#137437 [12:32:03] [ ⚓ T6956 Score Has Not Been Disabled ] - phabricator.miraheze.org [12:32:18] There's no information that we don't already know [12:32:24] And 0 risk [12:33:10] 👍 Just making sure you know. 🙂 [12:33:25] https://phabricator.miraheze.org/T5863 [12:33:26] [ ⚓ T5863 Extension:Score disabled due to multiple security issues ] - phabricator.miraheze.org [12:33:35] It says in the original task description [12:34:12] I don't get what is wrong with simply running a maint script though. [12:35:01] Argh... right, thanks. [12:35:15] Because currently the moment it's safe we can revert 2 commits and it will work again [12:35:44] If we disable everywhere we have to maintain a list and track where to re-enable on [12:36:56] Not that 8.5 months on I expect that to be anytime soon [12:49:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.71, 3.85, 3.99 [12:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.04, 3.85, 3.94 [12:53:30] Hi JohnLewis [12:55:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.33, 3.63, 3.85 [12:55:30] Hi [12:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.28, 4.28, 4.03 [13:21:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.46, 3.68, 4.00 [13:29:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.47, 3.96, 3.97 [13:33:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.32, 3.79, 3.92 [13:35:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.21, 3.98, 3.98 [13:47:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.86, 3.87, 3.98 [13:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.08, 3.88, 3.94 [14:13:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.93, 3.84, 3.97 [14:15:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.54, 4.06, 4.03 [14:39:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.76, 3.88, 3.98 [14:47:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.17, 4.03, 4.01 [14:49:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.70, 3.87, 3.95 [14:51:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.26, 4.05, 4.01 [15:19:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.05, 3.62, 3.97 [15:23:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.07, 3.66, 3.88 [15:25:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.66, 3.65, 3.85 [15:36:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCQY [15:36:53] [02miraheze/puppet] 07paladox 0353dc3cd - letsencrypt: Remove support for stretch [15:37:23] PROBLEM - www.burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Mon 29 Mar 2021 15:33:19 GMT +0000). [15:38:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCQc [15:38:22] [02miraheze/puppet] 07paladox 03748b932 - letsencrpt: Remove /etc/cron.d/certbot [15:39:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.14, 3.93, 3.84 [15:41:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.25, 3.83, 3.83 [15:45:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.01, 4.01, 3.90 [15:51:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.32, 3.88, 3.94 [16:01:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.46, 4.15, 3.99 [16:03:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.68, 3.98, 3.95 [16:05:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.34, 4.00, 3.95 [16:05:27] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 344.83 ms [16:07:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.47, 3.77, 3.87 [16:07:28] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 253.41 ms [16:11:51] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCds [16:11:53] [02miraheze/puppet] 07JohnFLewis 031b7d9bd - T6952: Varnish cleanup after failure [16:13:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCdl [16:13:20] [02miraheze/puppet] 07paladox 03f46e1f0 - letsencrypt: Ensure certbot and certbot.timer are stopped and masked [16:13:40] Who is responsible for the IR for the Varnish outage the other day? [16:14:08] [02puppet] 07dmehus commented on commit 031b7d9bd772c8d7f9a7474e0e909485e0807ca6c9 - 13https://git.io/JqCdR [16:14:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCdE [16:14:18] [02miraheze/puppet] 07paladox 031623a2b - Fix syntax [16:15:17] JohnLewis, no clue, but there should be an IR done, right? Does it have to be a Infra team member who completes it, or does Infra just have to review and approve it? [16:15:44] One has already been started, which is why I asked [16:15:49] oh [16:15:57] Infra didn't respond to it, so for Infra to do it would not be effective [16:16:05] true, yeah [16:17:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.18, 3.77, 3.76 [16:17:21] If my $0.02 cents (or pence) count, I'd probably recommend RhinosF1 for completing the IR as the first responding sysadmin. Reception123 responded as well later, so would be a logical choice, too [16:17:33] It's already been started [16:17:42] I just want to know who did it [16:17:46] oh [16:18:04] I think RhinosF1.. though not certain. [16:18:17] Reception123: ^^ [16:19:01] I don't know where they started it...latest IR modified was 7 March 2021 (https://meta.miraheze.org/wiki/Special:Log?type=incidentreporting&user=&page=&wpdate=&tagfilter=&wpfilters%5B%5D=newusers) [16:19:02] [ Incident report log - Miraheze Meta ] - meta.miraheze.org [16:19:22] Non published IRs aren't public [16:19:27] ah [16:19:36] okay, there's not even a log entry for it then? [16:19:44] No. [16:19:48] No, because it's not public [16:19:55] ah, okay [16:22:29] I didn't start it, so RhinosF1 must have then [16:23:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.43, 3.75, 3.76 [16:23:38] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCFs [16:23:39] [02miraheze/ssl] 07paladox 031d379e2 - Update dreamsit.com.br.crt [16:25:22] Okay, sent you a message [16:27:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.54, 3.84, 3.76 [16:32:23] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCFM [16:32:25] [02miraheze/ssl] 07paladox 031dc4b04 - Update www.burnout.wiki.crt [16:33:49] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCFd [16:33:50] [02miraheze/ssl] 07paladox 03e6852d0 - Allow events for reviwiki.info [16:34:27] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCFN [16:34:29] [02miraheze/ssl] 07paladox 036448dee - Update reviwiki.info.crt [16:36:31] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbU [16:36:32] [02miraheze/dns] 07paladox 03aa82993 - Add acme challenges to miraheze.wiki [16:37:44] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbL [16:37:46] [02miraheze/ssl] 07paladox 03e7d3164 - Update miraheze.wiki.crt [16:38:24] RECOVERY - www.burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.burnout.wiki' will expire on Fri 11 Jun 2021 15:31:25 GMT +0000. [16:39:10] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbm [16:39:12] [02miraheze/dns] 07paladox 030116753 - Add acme challenges to orain.org [16:39:40] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbG [16:39:41] [02miraheze/ssl] 07paladox 031c81834 - Update orain.org.crt [16:40:46] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 11 Jun 2021 15:39:25 GMT +0000. [16:40:50] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 11 Jun 2021 15:39:25 GMT +0000. [16:41:03] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 11 Jun 2021 15:39:25 GMT +0000. [16:41:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.48, 3.84, 3.94 [16:41:23] RECOVERY - bots1.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 11 Jun 2021 15:37:10 GMT +0000. [16:41:27] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Fri 11 Jun 2021 15:34:07 GMT +0000. [16:41:33] RECOVERY - tools1.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 11 Jun 2021 15:37:10 GMT +0000. [16:41:38] RECOVERY - db1.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 11 Jun 2021 15:37:10 GMT +0000. [16:41:42] RECOVERY - meta.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Fri 11 Jun 2021 15:39:25 GMT +0000. [16:41:51] RECOVERY - bots.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 11 Jun 2021 15:37:10 GMT +0000. [16:42:40] JohnLewis: me [16:42:46] I just need to add timings [16:42:47] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 11 Jun 2021 15:37:10 GMT +0000. [16:43:55] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbg [16:43:57] [02miraheze/ssl] 07paladox 03eb6cd02 - Enable events for mr.gyaanipedia.com [16:44:35] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbV [16:44:36] [02miraheze/ssl] 07paladox 0309e419e - Update mr.gyaanipedia.com.crt [16:44:54] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbo [16:44:56] [02miraheze/ssl] 07paladox 033f50fac - Enable events for pa.gyaanipedia.com [16:45:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbi [16:45:17] [02miraheze/ssl] 07paladox 034ff4b47 - Update pa.gyaanipedia.com.crt [16:45:17] I replied to Reception123 [16:45:47] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbX [16:45:49] [02miraheze/ssl] 07paladox 03751e852 - Enable events for hi.gyaanipedia.com [16:46:02] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCb1 [16:46:04] [02miraheze/ssl] 07paladox 037296409 - Update hi.gyaanipedia.com.crt [16:46:25] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbD [16:46:26] [02miraheze/ssl] 07paladox 03b6b34dd - Enable events for meta.gyaanipedia.com [16:46:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCb9 [16:46:50] [02miraheze/ssl] 07paladox 037a2af81 - Update meta.gyaanipedia.com.crt [16:47:13] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbH [16:47:15] [02miraheze/ssl] 07paladox 0357f28d3 - Enable events for commons.gyaanipedia.com [16:47:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCbQ [16:47:36] [02miraheze/ssl] 07paladox 03d9c7c3e - Update commons.gyaanipedia.com.crt [16:49:03] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqCbb [16:49:05] [02miraheze/ssl] 07paladox 0381763de - Remove virus23.uk.to [16:49:06] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:49:08] [02ssl] 07paladox opened pull request 03#397: Remove virus23.uk.to - 13https://git.io/JqCbN [16:50:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JqCbx [16:50:11] [02miraheze/ssl] 07paladox 03e9406a8 - Delete virus23.uk.to.crt [16:50:13] [02ssl] 07paladox synchronize pull request 03#397: Remove virus23.uk.to - 13https://git.io/JqCbN [16:50:17] [02ssl] 07paladox closed pull request 03#397: Remove virus23.uk.to - 13https://git.io/JqCbN [16:50:19] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JqCbh [16:50:20] [02miraheze/ssl] 07paladox 0339012cc - Remove virus23.uk.to (#397) [16:50:22] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:50:23] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [16:50:24] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:51:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.21, 3.82, 3.86 [16:51:07] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:52:21] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:53:48] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:55:03] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.88, 3.99, 3.93 [16:56:13] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [16:57:03] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 3.87, 4.01, 3.95 [16:58:13] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [17:02:50] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [17:03:20] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:47:22 GMT +0000. [17:03:55] RECOVERY - virus23.uk.to - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Sat 23 Oct 2021 23:59:59 GMT +0000. [17:03:55] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:04:10] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:04:11] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:44:14 GMT +0000. [17:04:13] RECOVERY - pa.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pa.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:45:02 GMT +0000. [17:04:17] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:45:41 GMT +0000. [17:04:20] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Fri 11 Jun 2021 15:46:34 GMT +0000. [17:05:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.37, 3.86, 3.95 [17:05:42] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCN6 [17:05:43] [02miraheze/dns] 07paladox 033ebb258 - Add acme challenges to miraheze.com [17:05:53] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [17:06:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCNi [17:06:18] [02miraheze/ssl] 07paladox 0321df6b6 - Update miraheze.com.crt [17:07:40] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqCND [17:07:41] [02miraheze/ssl] 07paladox 0320063df - Remove theatlas.pw [17:07:43] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:07:44] [02ssl] 07paladox opened pull request 03#398: Remove theatlas.pw - 13https://git.io/JqCNy [17:07:58] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JqCNS [17:07:59] [02miraheze/ssl] 07paladox 03e39c213 - Delete theatlas.pw.crt [17:08:01] [02ssl] 07paladox synchronize pull request 03#398: Remove theatlas.pw - 13https://git.io/JqCNy [17:08:07] [02ssl] 07paladox closed pull request 03#398: Remove theatlas.pw - 13https://git.io/JqCNy [17:08:09] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JqCN9 [17:08:10] [02miraheze/ssl] 07paladox 030d083a9 - Remove theatlas.pw (#398) [17:08:11] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:08:13] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:09:48] @JohnLewis or paladox: do you think you might be able to review/deploy https://github.com/miraheze/puppet/pull/17 and look into https://github.com/miraheze/puppet/pull/1703 when you have time? There's no hurry, especially with the second as that is definitely not high up on priority, but the first one will hopefully fix those index notices in search engines, so I'm hoping to get that deployed. But just when you have time, [17:09:49] both can wait longer, as it's been like that for months at least. [17:09:50] [ Add myself to icinga contacts by revi · Pull Request #17 · miraheze/puppet · GitHub ] - github.com [17:09:50] [ Allow wikis to control Robots.txt from MediaWiki:Robots.txt (T6881) by Universal-Omega · Pull Request #1703 · miraheze/puppet · GitHub ] - github.com [17:10:24] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqCNA [17:10:25] [02miraheze/ssl] 07paladox 03d0f97b1 - Remove wiki.iufs.jp [17:10:27] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:10:28] [02ssl] 07paladox opened pull request 03#399: Remove wiki.iufs.jp - 13https://git.io/JqCNp [17:10:39] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JqCNh [17:10:41] [02miraheze/ssl] 07paladox 034bdea86 - Delete wiki.iufs.jp.crt [17:10:42] [02ssl] 07paladox synchronize pull request 03#399: Remove wiki.iufs.jp - 13https://git.io/JqCNp [17:10:50] [02ssl] 07paladox closed pull request 03#399: Remove wiki.iufs.jp - 13https://git.io/JqCNp [17:10:52] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JqCAv [17:10:53] [02miraheze/ssl] 07paladox 03b3c7079 - Remove wiki.iufs.jp (#399) [17:10:55] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:10:56] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:11:12] Universal_Omega: you probably pasted wrong link for the first one? [17:11:18] merged 2015 [17:11:50] Oh whoops. Checking. [17:11:54] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Fri 11 Jun 2021 16:05:57 GMT +0000. [17:11:59] Yeah, sorry. [17:12:10] How did that happen? [17:13:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.39, 4.12, 4.01 [17:13:10] revi: https://github.com/miraheze/puppet/pull/1702 (must've accidentally erased last 2 digits) [17:13:11] [ Varnish: Add X-Robots-Tag http header (Should fix T6680) by Universal-Omega · Pull Request #1702 · miraheze/puppet · GitHub ] - github.com [17:13:47] Universal_Omega: You can also ask Reception as they're an SRE member too [17:15:08] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JqCA3 [17:15:09] [02miraheze/ssl] 07paladox 039d5bc66 - Remove some domains that are not pointing at us [17:15:11] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:15:12] [02ssl] 07paladox opened pull request 03#400: Remove some domains that are not pointing at us - 13https://git.io/JqCAs [17:15:29] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JqCAG [17:15:31] [02miraheze/ssl] 07paladox 03eddb205 - Delete duepedia.uk.to.crt [17:15:32] [02ssl] 07paladox synchronize pull request 03#400: Remove some domains that are not pointing at us - 13https://git.io/JqCAs [17:15:40] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JqCAZ [17:15:42] [02miraheze/ssl] 07paladox 03119b2ae - Delete privacy-wiki.0x.no.crt [17:15:43] [02ssl] 07paladox synchronize pull request 03#400: Remove some domains that are not pointing at us - 13https://git.io/JqCAs [17:16:30] [02ssl] 07paladox closed pull request 03#400: Remove some domains that are not pointing at us - 13https://git.io/JqCAs [17:16:32] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-2/±1] 13https://git.io/JqCA8 [17:16:33] [02miraheze/ssl] 07paladox 03eeb6462 - Remove some domains that are not pointing at us (#400) [17:16:35] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [17:16:36] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [17:19:05] JohnLewis: I know, but I thought it'd be better for infrastructure to review, the Varnish one especially. [17:21:12] Reception is probably able to review it quicker and with the appropriate priority though, especially if the tasks they're working on are MW-SRE ones [17:21:23] seems fine, though preferably Reception123 should have the sign off as the MW Manger [17:24:43] [02puppet] 07paladox closed pull request 03#1702: Varnish: Add X-Robots-Tag http header (Should fix T6680) - 13https://git.io/JqZ7S [17:24:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCAo [17:24:46] [02miraheze/puppet] 07Universal-Omega 033f2ee04 - Varnish: Add X-Robots-Tag http header (Should fix T6680) (#1702) [17:25:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 6.53, 5.10 [17:25:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.13, 6.54, 5.15 [17:27:14] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.39, 5.88, 5.03 [17:27:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.91, 6.36, 5.25 [17:28:57] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.52, 3.79, 3.97 [17:29:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.02, 8.14, 6.04 [17:30:56] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.32, 4.02, 4.03 [17:31:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.10, 7.54, 6.05 [17:33:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.92, 8.54, 6.58 [17:35:31] PROBLEM - christipedia.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for christipedia.nl could not be found [17:35:32] PROBLEM - www.wiki.mirahezebots.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wiki.mirahezebots.org could not be found [17:35:33] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [17:35:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.30, 8.35, 6.48 [17:36:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.18, 7.65, 6.23 [17:36:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.10, 7.51, 6.03 [17:36:55] .ping meta.miraheze.org [17:36:55] Pong! [17:37:23] paladox: things seem super sluggish [17:38:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.04, 7.90, 6.36 [17:39:23] JohnLewis: ^ [17:39:24] Seems higher traffic is the cause [17:39:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.31, 8.00, 7.06 [17:39:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.46, 7.86, 6.71 [17:39:57] We've been discussing this [17:40:24] Where? [17:40:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.15, 7.38, 6.36 [17:41:11] Paladox PM'd me [17:41:30] Ah ok [17:41:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.71, 6.34, 6.29 [17:41:50] That level of slowness I just got was way too high [17:41:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.38, 5.81, 5.97 [17:42:01] Something needs to be done [17:42:14] RECOVERY - christipedia.nl - reverse DNS on sslhost is OK: rDNS OK - christipedia.nl reverse DNS resolves to cp10.miraheze.org [17:42:19] RECOVERY - www.wiki.mirahezebots.org - reverse DNS on sslhost is OK: rDNS OK - www.wiki.mirahezebots.org reverse DNS resolves to cp10.miraheze.org [17:42:21] RECOVERY - wiki.climatechange.ai - reverse DNS on sslhost is OK: rDNS OK - wiki.climatechange.ai reverse DNS resolves to cp11.miraheze.org [17:42:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.48, 6.20, 6.04 [17:43:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.11, 5.88, 6.41 [17:54:01] paladox: I think I messed up on regex in https://github.com/miraheze/puppet/pull/1702, the header is only added for the exact url, so it is for example added for https://meta.miraheze.org/wiki/Special: but not for https://meta.miraheze.org/wiki/Special:SpecialPages, any idea what I did wrong? [17:54:02] [ Varnish: Add X-Robots-Tag http header (Should fix T6680) by Universal-Omega · Pull Request #1702 · miraheze/puppet · GitHub ] - github.com [17:54:03] [ Special - Miraheze Meta ] - meta.miraheze.org [17:54:05] [ Special pages - Miraheze Meta ] - meta.miraheze.org [17:56:49] You need to add .+ after it [17:57:13] Oh. [17:58:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 6.09, 5.51 [17:59:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.58, 7.44, 6.41 [18:00:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.04, 5.25, 5.28 [18:01:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.62, 6.36, 6.14 [18:10:05] [02puppet] 07Universal-Omega opened pull request 03#1705: Fix X-Robots-Tag - 13https://git.io/JqCpM [18:11:40] paladox: ^ [18:28:55] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCh1 [18:28:56] [02miraheze/landing] 07Universal-Omega 03e16ec5d - Don't load qqq language code [18:29:55] miraheze/landing - Universal-Omega the build passed. [18:52:30] !log GDPR script [18:52:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:05] [02puppet] 07Universal-Omega synchronize pull request 03#1705: Fix X-Robots-Tag - 13https://git.io/JqCpM [19:04:25] Reception123: Do you mind merging ^ and taking a look at https://github.com/miraheze/puppet/pull/1703 when you have time ? [19:04:26] [ Allow wikis to control Robots.txt from MediaWiki:Robots.txt (T6881) by Universal-Omega · Pull Request #1703 · miraheze/puppet · GitHub ] - github.com [19:12:04] [02puppet] 07Universal-Omega synchronize pull request 03#1703: Allow wikis to control Robots.txt from MediaWiki:Robots.txt (T6881) - 13https://git.io/JqZjt [19:13:07] [02puppet] 07Reception123 closed pull request 03#1705: Fix X-Robots-Tag - 13https://git.io/JqCpM [19:13:09] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWv8 [19:13:10] [02miraheze/puppet] 07Universal-Omega 03643c47b - Fix X-Robots-Tag (#1705) [19:13:22] Universal_Omega: the latter I'd like paladox to review first [19:19:27] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.07, 3.57, 3.95 [19:19:59] [02puppet] 07paladox reviewed pull request 03#1705 commit - 13https://git.io/JqWvM [19:22:45] [02puppet] 07Universal-Omega reviewed pull request 03#1705 commit - 13https://git.io/JqWvH [19:23:02] [02puppet] 07Universal-Omega reviewed pull request 03#1705 commit - 13https://git.io/JqWvH [19:25:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.29, 3.87, 3.94 [19:35:01] PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.05, 5.44, 4.63 [19:36:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.00, 5.71 [19:38:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 17.80, 9.50, 6.69 [19:42:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 3.54, 6.99, 6.42 [19:43:01] RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 3.06, 5.04, 4.91 [19:43:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.65, 3.75, 3.95 [19:44:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.20, 5.73, 6.02 [19:47:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.28, 3.84, 3.92 [19:50:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWJe [19:50:39] [02miraheze/puppet] 07paladox 032a07a01 - dns: Remove support for stretch [19:51:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.51, 3.85, 3.93 [19:51:30] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 17 minutes ago with 0 failures [19:53:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWJt [19:53:13] [02miraheze/puppet] 07paladox 0387a43fd - dns: Notify gdnsd service when updating geoip file [19:53:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 6.11, 4.50, 4.15 [19:55:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWJZ [19:55:39] [02miraheze/puppet] 07paladox 03d73b7d4 - Fix [19:55:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:59:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWJR [19:59:37] [02miraheze/puppet] 07paladox 0337a3ae5 - Fix [19:59:49] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:59:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:05:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.96, 3.84, 3.99 [20:11:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWUq [20:11:02] [02miraheze/mw-config] 07Universal-Omega 031389614 - Update ManageWikiSettings.php [20:11:04] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [20:12:04] miraheze/mw-config - Universal-Omega the build passed. [20:12:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWUO [20:12:18] [02miraheze/mw-config] 07Universal-Omega 035046866 - Update LocalSettings.php [20:12:20] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [20:13:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.16, 3.80, 3.89 [20:13:24] miraheze/mw-config - Universal-Omega the build passed. [20:15:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.97, 3.47, 3.76 [20:20:56] [02CreateWiki] 07Reception123 closed pull request 03#200: AI Implementation - 13https://git.io/JqZaH [20:20:58] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JqWUV [20:20:59] [02miraheze/CreateWiki] 07JohnFLewis 0329279f5 - AI Implementation (#200) [20:21:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.20, 3.94, 3.88 [20:21:37] (FYI this is just merging it, not actually implementing it) [20:21:54] miraheze/CreateWiki - Reception123 the build passed. [20:22:16] RhinosF1: yo [20:22:46] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03aiimplement [20:22:47] [02CreateWiki] 07Universal-Omega deleted branch 03aiimplement - 13https://git.io/vpJTL [20:23:18] Cyberpower678: we have had a lot of load issues and IA bot is a very high requestor. paladox is looking at figures but it's been enough to cause significantly noticeable latency. [20:23:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.39, 3.75, 3.81 [20:23:49] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:04] RhinosF1: was it a spike in requests at a given interval>? [20:24:32] paladox: can you paste the stats you just gave ^ [20:24:35] Cyberpower678 i'm wondering if you can possibly slow your bot down please? From the logs i see 114104 requests today from IA. It is the one that stood out when i chcked. [20:26:23] paladox: my bot isn't running particularly fast on Miraheze. Is there a spike of requests in the logs. If so, that may have been resolved very recently. [20:27:09] But it has no operational throttle. It's just a single-process application. [20:27:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.22, 3.88, 3.85 [20:27:30] It has a rate-limiter for edits though. [20:27:40] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqWUx [20:27:41] [02miraheze/mediawiki] 07Reception123 03c744a2e - Update CreateWiki [20:28:13] i mean from here https://grafana.miraheze.org/d/LaSguOMZz/varnish?orgId=1&var-varnish_instance=All we see a spike, which correlated to the high load on our mediawiki servers. When checking the logs IA came out as noticable. Its possible that someone did a large amount of requests during a short period of time and wouldn't have been detected when i checked. [20:28:14] [ Grafana ] - grafana.miraheze.org [20:30:07] I really couldn't say what that is from. But it's not possible to throttle down the bot. [20:35:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.83, 3.97, 3.95 [20:39:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.06, 3.93, 3.93 [20:47:24] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.32, 3.75, 3.89 [20:51:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWkU [20:51:21] [02miraheze/mw-config] 07Universal-Omega 03eeb4c01 - Update ManageWikiSettings.php [20:51:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.04, 3.86, 3.90 [20:51:23] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [20:52:19] miraheze/mw-config - Universal-Omega the build passed. [20:53:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.94, 3.86, 3.90 [20:54:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWkG [20:54:23] [02miraheze/mw-config] 07Universal-Omega 0367e77c9 - Update ManageWikiSettings.php [20:54:24] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [20:55:24] miraheze/mw-config - Universal-Omega the build passed. [20:57:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWku [20:57:36] [02miraheze/mw-config] 07Universal-Omega 0370d117f - Update ManageWikiSettings.php [20:57:38] [02mw-config] 07Universal-Omega synchronize pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [20:58:39] miraheze/mw-config - Universal-Omega the build passed. [20:59:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.46, 4.38, 4.08 [21:08:39] [02mw-config] 07Universal-Omega closed pull request 03#3779: Add validation for shared uploads - 13https://git.io/JqC0g [21:08:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JqWIg [21:08:42] [02miraheze/mw-config] 07Universal-Omega 03000a206 - Add validation for shared uploads (#3779) [21:08:44] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:08:45] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:09:39] miraheze/mw-config - Universal-Omega the build passed. [21:46:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWtW [21:46:51] [02miraheze/mw-config] 07Universal-Omega 038601e66 - Move remaining configs out of LocalExtensions.php [21:46:53] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [21:46:54] [02mw-config] 07Universal-Omega opened pull request 03#3780: Move remaining configs out of LocalExtensions.php - 13https://git.io/JqWtl [21:47:54] miraheze/mw-config - Universal-Omega the build passed. [21:49:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWtr [21:49:19] [02miraheze/mw-config] 07Universal-Omega 03939b9be - Update LocalExtensions.php [21:49:21] [02mw-config] 07Universal-Omega synchronize pull request 03#3780: Move remaining configs out of LocalExtensions.php - 13https://git.io/JqWtl [21:50:19] miraheze/mw-config - Universal-Omega the build passed. [21:52:30] [02mw-config] 07Universal-Omega synchronize pull request 03#3780: Move remaining configs out of LocalExtensions.php - 13https://git.io/JqWtl [21:52:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWty [21:52:33] [02miraheze/mw-config] 07Universal-Omega 03e5a6914 - Update LocalSettings.php [21:53:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWtQ [21:53:23] [02miraheze/mw-config] 07Universal-Omega 03ba776f0 - Update LocalSettings.php [21:53:24] [02mw-config] 07Universal-Omega synchronize pull request 03#3780: Move remaining configs out of LocalExtensions.php - 13https://git.io/JqWtl [21:53:34] miraheze/mw-config - Universal-Omega the build passed. [21:54:24] miraheze/mw-config - Universal-Omega the build passed. [21:55:34] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:57:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.51, 3.75, 3.99 [21:59:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.21, 3.98, 4.05 [22:03:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.68, 3.86, 3.99 [22:05:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.23, 3.97, 4.02 [22:07:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.68, 3.88, 3.98 [22:09:42] [02mw-config] 07Universal-Omega closed pull request 03#3780: Move remaining configs out of LocalExtensions.php - 13https://git.io/JqWtl [22:09:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqWqR [22:09:45] [02miraheze/mw-config] 07Universal-Omega 039631057 - Move remaining configs out of LocalExtensions.php (#3780) [22:09:47] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [22:09:48] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:10:44] miraheze/mw-config - Universal-Omega the build passed. [22:15:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWqi [22:15:53] [02miraheze/mw-config] 07Universal-Omega 03a8b3a0f - Move config to Wikibase.php [22:16:49] miraheze/mw-config - Universal-Omega the build passed. [22:16:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWq1 [22:16:55] [02miraheze/mw-config] 07Universal-Omega 03dd670c9 - Move config to Wikibase.php [22:17:49] miraheze/mw-config - Universal-Omega the build passed. [22:18:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWqM [22:18:10] [02miraheze/mw-config] 07Universal-Omega 0356a5258 - Fix $wmgVisualEditorEnableDefault [22:19:04] miraheze/mw-config - Universal-Omega the build passed. [22:21:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqWqd [22:21:05] [02miraheze/mw-config] 07Universal-Omega 0305a5bd1 - add newline for formatting [22:21:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.25, 3.98, 3.84 [22:22:04] miraheze/mw-config - Universal-Omega the build passed. [22:35:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.76, 6.91, 5.58 [22:35:28] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.69, 3.95, 3.97 [22:36:49] PROBLEM - enc.for.uz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for enc.for.uz could not be found [22:36:56] PROBLEM - miraheze.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.tk could not be found [22:37:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.74, 6.53, 5.60 [22:37:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.06, 4.03, 4.00 [22:37:42] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [22:37:56] Universal_Omega: ^ [22:39:26] RhinosF1: I know, been testing stuff on test3. [22:39:37] Ack cool [22:41:09] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.52, 7.18, 6.17 [22:41:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.74, 7.24, 6.02 [22:41:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.93, 7.08, 5.86 [22:43:37] RECOVERY - enc.for.uz - reverse DNS on sslhost is OK: rDNS OK - enc.for.uz reverse DNS resolves to cp11.miraheze.org [22:43:48] RECOVERY - miraheze.tk - reverse DNS on sslhost is OK: rDNS OK - miraheze.tk reverse DNS resolves to cp10.miraheze.org [22:45:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.18, 5.70, 5.83 [22:45:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.13, 5.98, 5.79 [22:45:30] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.14, 3.78, 3.94 [22:45:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 6.07, 5.74 [22:48:27] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.37, 7.34, 6.21 [22:48:33] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [22:48:43] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for adadevelopersacademy.wiki could not be found [22:48:46] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [22:48:46] PROBLEM - intp.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for intp.miraheze.org could not be found [22:48:48] PROBLEM - kk.uncyc.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kk.uncyc.tk could not be found [22:49:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.31, 6.96, 6.29 [22:49:21] Universa_Omega: Moving everything back to LS undoes all the work we’ve done over the years to reduce the size of LS and simply the configuration file [22:49:27] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.70, 4.01, 3.98 [22:49:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.36, 7.20, 6.35 [22:49:34] * Universal_Omega [22:51:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.65, 7.68, 6.61 [22:51:25] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.85, 3.99, 3.99 [22:51:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.12, 7.92, 6.70 [22:53:21] PROBLEM - pubwiki.lab.co.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pubwiki.lab.co.pl could not be found [22:53:23] PROBLEM - ipv6bolivia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ipv6bolivia.tk could not be found [22:53:23] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [22:53:23] PROBLEM - nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nenawiki.org could not be found [22:53:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.71, 7.53, 6.68 [22:55:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.59, 6.25, 6.32 [22:55:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.59, 6.99, 6.67 [22:55:31] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - hi.gyaanipedia.com reverse DNS resolves to cp11.miraheze.org [22:55:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.16, 6.21, 6.30 [22:55:33] RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: rDNS OK - adadevelopersacademy.wiki reverse DNS resolves to cp11.miraheze.org [22:55:38] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: rDNS OK - wiki.zymonic.com reverse DNS resolves to cp10.miraheze.org [22:55:38] RECOVERY - intp.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - intp.miraheze.org reverse DNS resolves to cp11.miraheze.org [22:55:42] RECOVERY - kk.uncyc.tk - reverse DNS on sslhost is OK: rDNS OK - kk.uncyc.tk reverse DNS resolves to cp11.miraheze.org [22:56:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.60, 6.92, 6.66 [22:57:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.56, 5.78, 6.26 [22:58:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.62, 5.86, 6.30 [22:59:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.34, 3.81, 3.86 [23:00:09] [02mw-config] 07JohnFLewis commented on pull request 03#3780: Move remaining configs out of LocalExtensions.php - 13https://git.io/JqWYl [23:00:12] RECOVERY - pubwiki.lab.co.pl - reverse DNS on sslhost is OK: rDNS OK - pubwiki.lab.co.pl reverse DNS resolves to cp11.miraheze.org [23:00:15] RECOVERY - ipv6bolivia.tk - reverse DNS on sslhost is OK: rDNS OK - ipv6bolivia.tk reverse DNS resolves to cp10.miraheze.org [23:00:16] RECOVERY - nenawiki.org - reverse DNS on sslhost is OK: rDNS OK - nenawiki.org reverse DNS resolves to cp10.miraheze.org [23:00:17] RECOVERY - wiki.shaazzz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.shaazzz.ir reverse DNS resolves to cp10.miraheze.org [23:01:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.57, 3.82, 3.87 [23:03:43] JohnLewis: I only added configs to the bottom of LS, which could not be done via the method that you created to do that with https://github.com/miraheze/CreateWiki/commit/356abb5, which I finished implementing. I do still plan to cleanup the entirety of LocalSettings.php, and LocalWiki.php, in order to simplify and clean things up, right now I am just trying to finish the work you started and completely get rid of [23:03:43] LocalExtensions.php, but if you want me to stop and undo what I have done, then I will do that. What I wanted to do eventually though was simplify everything through using another file (or maybe not, just LocalWiki.php is used for to much) for functions and custom miraheze configs setting (such as https://github.com/miraheze/mw-config/blob/05a5bd1/LocalSettings.php#L4317-L4329). But I can undo it now, if you want, and [23:03:43] re-implement it once I have a better method to do so, which admittedly I should've went about that a different way. [23:03:44] [ Add Extensions to a SiteConfiguration Object Tag · miraheze/CreateWiki@356abb5 · GitHub ] - github.com [23:03:44] [ mw-config/LocalSettings.php at 05a5bd1c49a394d930fb15def1598be46c9a76ad · miraheze/mw-config · GitHub ] - github.com [23:05:08] The issue is we have had a massive push for moving stuff away from LS, but without discussion with anyone else you’ve decided to reverse our objectives and push everything back into LS [23:06:27] If they can’t be added to the right place in LS, adding them to the bottom of the file isn’t an appropriate alternative as we’ve spent years to get to the point where we can remove the edge cases out of the bottom of the file [23:09:47] JohnLewis: Fair enough. I will remove them from the bottom of LS.php and add back into LE.php, which I do understand should have been done from the beginning to just leave them there. But I was trying to finish the work you started with LE, but I guess I should not have done this yet. I see your point, and apologise for undoing the work, that was not my intention. [23:11:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.09, 3.68, 3.75 [23:11:58] The work needed to do what you want is far from done. There were hours of coding left to be done to achieve the system i was working on, which I have no since abandoned when stepping back. The only code published is minimalist and it’s not possible to convert based on it [23:13:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.24, 3.51, 3.68 [23:13:23] what is this https://github.com/miraheze/mw-config/pull/3780/files#diff-35b93cff8d87232a32e7a37a41c1e96682b8c911f123f1ffec8f3291b5bbfdb0R3974 [23:13:23] [ Move remaining configs out of LocalExtensions.php by Universal-Omega · Pull Request #3780 · miraheze/mw-config · GitHub ] - github.com [23:16:20] My advice would be work through tasks, as there’s a lot of long running large project tasks - ones which provide immense more value than a file rearrangement. Two of which are goals due by June as well [23:16:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWOk [23:16:57] [02miraheze/mw-config] 07Universal-Omega 03590f4d5 - Add configs back to LocalExtensions.php [23:16:58] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [23:17:00] [02mw-config] 07Universal-Omega opened pull request 03#3781: Add configs back to LocalExtensions.php - 13https://git.io/JqWOI [23:17:59] miraheze/mw-config - Universal-Omega the build passed. [23:18:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqWOY [23:18:14] [02miraheze/mw-config] 07Universal-Omega 03d3bd549 - Update LocalSettings.php [23:18:16] [02mw-config] 07Universal-Omega synchronize pull request 03#3781: Add configs back to LocalExtensions.php - 13https://git.io/JqWOI [23:19:14] miraheze/mw-config - Universal-Omega the build passed. [23:22:21] [02mw-config] 07Universal-Omega closed pull request 03#3781: Add configs back to LocalExtensions.php - 13https://git.io/JqWOI [23:22:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqWOR [23:22:24] [02miraheze/mw-config] 07Universal-Omega 0388cadd3 - Add configs back to LocalExtensions.php (#3781) [23:22:25] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [23:22:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:23:19] miraheze/mw-config - Universal-Omega the build passed. [23:26:01] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:47] @JohnLewis: those goal tasks are likely beyond my experience, but yes, I will stick to tasks, and not make pointless changes in the future. [23:29:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.01, 3.96, 3.79 [23:34:06] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 174s [23:36:06] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [23:39:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.48, 3.88, 3.89 [23:42:06] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 128s [23:42:48] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.86, 3.44, 2.30 [23:43:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.08, 3.98, 3.92 [23:44:09] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 19s [23:44:48] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.54, 2.82, 2.22 [23:45:22] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.32, 3.74, 3.84 [23:49:23] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.42, 4.23, 3.99 [23:54:06] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 281s