[00:04:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.03, 7.33, 6.52 [00:05:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 6.94, 8.27, 7.15 [00:06:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.88, 7.95, 6.88 [00:08:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Citizen-b625a5f [+0/-0/±1] 13https://git.io/JYSFH [00:08:25] [02miraheze/mediawiki] 07dependabot[bot] 035971548 - Bump skins/Citizen from `2653336` to `b625a5f` [00:08:27] [02mediawiki] 07dependabot[bot] opened pull request 03#1403: Bump skins/Citizen from `2653336` to `b625a5f` - 13https://git.io/JYSFQ [00:08:28] [02mediawiki] 07dependabot[bot] labeled pull request 03#1403: Bump skins/Citizen from `2653336` to `b625a5f` - 13https://git.io/JYSFQ [00:08:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#1403: Bump skins/Citizen from `2653336` to `b625a5f` - 13https://git.io/JYSFQ [00:08:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Citizen-b625a5f - 13https://git.io/vbL5b [00:08:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#1401: Bump skins/Citizen from `2653336` to `215fb5f` - 13https://git.io/JYSF7 [00:08:34] [02mediawiki] 07dependabot[bot] closed pull request 03#1401: Bump skins/Citizen from `2653336` to `215fb5f` - 13https://git.io/JYcAK [00:08:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-215fb5f - 13https://git.io/vbL5b [00:08:37] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-215fb5f [00:08:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ElectronPdfService-0d21aed [+0/-0/±1] 13https://git.io/JYSF5 [00:08:40] [02miraheze/mediawiki] 07dependabot[bot] 039cc0752 - Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` [00:08:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ElectronPdfService-0d21aed - 13https://git.io/vbL5b [00:08:43] [02mediawiki] 07dependabot[bot] opened pull request 03#1404: Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` - 13https://git.io/JYSFd [00:08:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#1404: Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` - 13https://git.io/JYSFd [00:08:46] [02mediawiki] 07dependabot[bot] labeled pull request 03#1404: Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` - 13https://git.io/JYSFd [00:08:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/WikiSEO-c4a44ea [+0/-0/±1] 13https://git.io/JYSFF [00:08:49] [02miraheze/mediawiki] 07dependabot[bot] 03a7bdc0d - Bump extensions/WikiSEO from `c0579da` to `c4a44ea` [00:08:51] [02mediawiki] 07dependabot[bot] opened pull request 03#1405: Bump extensions/WikiSEO from `c0579da` to `c4a44ea` - 13https://git.io/JYSFb [00:08:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-c4a44ea - 13https://git.io/vbL5b [00:08:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1405: Bump extensions/WikiSEO from `c0579da` to `c4a44ea` - 13https://git.io/JYSFb [00:08:55] [02mediawiki] 07dependabot[bot] labeled pull request 03#1405: Bump extensions/WikiSEO from `c0579da` to `c4a44ea` - 13https://git.io/JYSFb [00:08:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#1385: Bump extensions/WikiSEO from `c0579da` to `1e8a8ba` - 13https://git.io/JYSFN [00:08:58] [02mediawiki] 07dependabot[bot] closed pull request 03#1385: Bump extensions/WikiSEO from `c0579da` to `1e8a8ba` - 13https://git.io/JYcNx [00:09:00] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-1e8a8ba [00:09:01] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-1e8a8ba - 13https://git.io/vbL5b [00:09:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/LinkTitles-4a2d7c9 [+0/-0/±1] 13https://git.io/JYSFA [00:09:04] [02miraheze/mediawiki] 07dependabot[bot] 030afaae4 - Bump extensions/LinkTitles from `a77bf03` to `4a2d7c9` [00:09:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/LinkTitles-4a2d7c9 - 13https://git.io/vbL5b [00:09:07] [02mediawiki] 07dependabot[bot] opened pull request 03#1406: Bump extensions/LinkTitles from `a77bf03` to `4a2d7c9` - 13https://git.io/JYSFx [00:09:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1406: Bump extensions/LinkTitles from `a77bf03` to `4a2d7c9` - 13https://git.io/JYSFx [00:09:10] [02mediawiki] 07dependabot[bot] labeled pull request 03#1406: Bump extensions/LinkTitles from `a77bf03` to `4a2d7c9` - 13https://git.io/JYSFx [00:09:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/AJAXPoll-4ab46c2 [+0/-0/±1] 13https://git.io/JYSFh [00:09:13] [02miraheze/mediawiki] 07dependabot[bot] 03cfff408 - Bump extensions/AJAXPoll from `3830cb4` to `4ab46c2` [00:09:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-4ab46c2 - 13https://git.io/vbL5b [00:09:16] [02mediawiki] 07dependabot[bot] opened pull request 03#1407: Bump extensions/AJAXPoll from `3830cb4` to `4ab46c2` - 13https://git.io/JYSFj [00:09:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#1407: Bump extensions/AJAXPoll from `3830cb4` to `4ab46c2` - 13https://git.io/JYSFj [00:09:19] [02mediawiki] 07dependabot[bot] labeled pull request 03#1407: Bump extensions/AJAXPoll from `3830cb4` to `4ab46c2` - 13https://git.io/JYSFj [00:09:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#1386: Bump extensions/AJAXPoll from `3830cb4` to `e0ca44f` - 13https://git.io/JYSbe [00:09:22] [02mediawiki] 07dependabot[bot] closed pull request 03#1386: Bump extensions/AJAXPoll from `3830cb4` to `e0ca44f` - 13https://git.io/JYcNj [00:09:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-e0ca44f - 13https://git.io/vbL5b [00:09:25] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-e0ca44f [00:09:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/RevisionSlider-d1a6af2 [+0/-0/±1] 13https://git.io/JYSbv [00:09:28] [02miraheze/mediawiki] 07dependabot[bot] 038483f59 - Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` [00:09:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/RevisionSlider-d1a6af2 - 13https://git.io/vbL5b [00:09:31] [02mediawiki] 07dependabot[bot] opened pull request 03#1408: Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` - 13https://git.io/JYSbf [00:09:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1408: Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` - 13https://git.io/JYSbf [00:09:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#1408: Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` - 13https://git.io/JYSbf [00:09:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/BlogPage-140883c [+0/-0/±1] 13https://git.io/JYSbJ [00:09:37] [02miraheze/mediawiki] 07dependabot[bot] 037a68f7c - Bump extensions/BlogPage from `1e366df` to `140883c` [00:09:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/BlogPage-140883c - 13https://git.io/vbL5b [00:09:40] [02mediawiki] 07dependabot[bot] opened pull request 03#1409: Bump extensions/BlogPage from `1e366df` to `140883c` - 13https://git.io/JYSbU [00:09:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#1409: Bump extensions/BlogPage from `1e366df` to `140883c` - 13https://git.io/JYSbU [00:09:43] [02mediawiki] 07dependabot[bot] labeled pull request 03#1409: Bump extensions/BlogPage from `1e366df` to `140883c` - 13https://git.io/JYSbU [00:09:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#1387: Bump extensions/BlogPage from `1e366df` to `beb02b8` - 13https://git.io/JYSbT [00:09:46] [02mediawiki] 07dependabot[bot] closed pull request 03#1387: Bump extensions/BlogPage from `1e366df` to `beb02b8` - 13https://git.io/JYcAv [00:09:48] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/BlogPage-beb02b8 [00:09:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 7.43, 7.09 [00:09:49] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/BlogPage-beb02b8 - 13https://git.io/vbL5b [00:09:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Cargo-d31ffcc [+0/-0/±1] 13https://git.io/JYSbk [00:09:52] [02miraheze/mediawiki] 07dependabot[bot] 03a3497a2 - Bump extensions/Cargo from `8fc3204` to `d31ffcc` [00:09:54] [02mediawiki] 07dependabot[bot] opened pull request 03#1410: Bump extensions/Cargo from `8fc3204` to `d31ffcc` - 13https://git.io/JYSbI [00:09:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Cargo-d31ffcc - 13https://git.io/vbL5b [00:09:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#1410: Bump extensions/Cargo from `8fc3204` to `d31ffcc` - 13https://git.io/JYSbI [00:09:58] [02mediawiki] 07dependabot[bot] labeled pull request 03#1410: Bump extensions/Cargo from `8fc3204` to `d31ffcc` - 13https://git.io/JYSbI [00:10:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#1393: Bump extensions/Cargo from `8fc3204` to `e027884` - 13https://git.io/JYSbL [00:10:01] [02mediawiki] 07dependabot[bot] closed pull request 03#1393: Bump extensions/Cargo from `8fc3204` to `e027884` - 13https://git.io/JYcAn [00:10:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Cite-bc16b05 [+0/-0/±1] 13https://git.io/JYSbt [00:10:04] [02miraheze/mediawiki] 07dependabot[bot] 0348fbb9a - Bump extensions/Cite from `52f3cf3` to `bc16b05` [00:10:06] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Cargo-e027884 - 13https://git.io/vbL5b [00:10:07] [02mediawiki] 07dependabot[bot] opened pull request 03#1411: Bump extensions/Cite from `52f3cf3` to `bc16b05` - 13https://git.io/JYSbq [00:10:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Cite-bc16b05 - 13https://git.io/vbL5b [00:10:10] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Cargo-e027884 [00:10:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#1411: Bump extensions/Cite from `52f3cf3` to `bc16b05` - 13https://git.io/JYSbq [00:10:13] [02mediawiki] 07dependabot[bot] labeled pull request 03#1411: Bump extensions/Cite from `52f3cf3` to `bc16b05` - 13https://git.io/JYSbq [00:10:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PageForms-d7f5a20 [+0/-0/±1] 13https://git.io/JYSbm [00:10:17] [02miraheze/mediawiki] 07dependabot[bot] 0389c3afc - Bump extensions/PageForms from `ff424c0` to `d7f5a20` [00:10:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PageForms-d7f5a20 - 13https://git.io/vbL5b [00:10:19] [02mediawiki] 07dependabot[bot] opened pull request 03#1412: Bump extensions/PageForms from `ff424c0` to `d7f5a20` - 13https://git.io/JYSbY [00:10:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#1412: Bump extensions/PageForms from `ff424c0` to `d7f5a20` - 13https://git.io/JYSbY [00:10:23] [02mediawiki] 07dependabot[bot] labeled pull request 03#1412: Bump extensions/PageForms from `ff424c0` to `d7f5a20` - 13https://git.io/JYSbY [00:10:24] [02mediawiki] 07dependabot[bot] commented on pull request 03#1399: Bump extensions/PageForms from `ff424c0` to `0a566aa` - 13https://git.io/JYSbO [00:10:26] [02mediawiki] 07dependabot[bot] closed pull request 03#1399: Bump extensions/PageForms from `ff424c0` to `0a566aa` - 13https://git.io/JYcAV [00:10:27] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-0a566aa - 13https://git.io/vbL5b [00:10:29] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-0a566aa [00:10:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/TwoColConflict-29639ad [+0/-0/±1] 13https://git.io/JYSbs [00:10:32] [02miraheze/mediawiki] 07dependabot[bot] 03637f324 - Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` [00:10:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/TwoColConflict-29639ad - 13https://git.io/vbL5b [00:10:35] [02mediawiki] 07dependabot[bot] opened pull request 03#1413: Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` - 13https://git.io/JYSbG [00:10:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#1413: Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` - 13https://git.io/JYSbG [00:10:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#1413: Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` - 13https://git.io/JYSbG [00:10:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/RelatedArticles-954f1b6 [+0/-0/±1] 13https://git.io/JYSbZ [00:10:41] [02miraheze/mediawiki] 07dependabot[bot] 03f645812 - Bump extensions/RelatedArticles from `8a38280` to `954f1b6` [00:10:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/RelatedArticles-954f1b6 - 13https://git.io/vbL5b [00:10:44] [02mediawiki] 07dependabot[bot] opened pull request 03#1414: Bump extensions/RelatedArticles from `8a38280` to `954f1b6` - 13https://git.io/JYSbn [00:10:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#1414: Bump extensions/RelatedArticles from `8a38280` to `954f1b6` - 13https://git.io/JYSbn [00:10:47] [02mediawiki] 07dependabot[bot] labeled pull request 03#1414: Bump extensions/RelatedArticles from `8a38280` to `954f1b6` - 13https://git.io/JYSbn [00:10:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-cb4c7a1 [+0/-0/±1] 13https://git.io/JYSbc [00:10:50] [02miraheze/mediawiki] 07dependabot[bot] 038959e5c - Bump extensions/UniversalLanguageSelector from `f31b0c9` to `cb4c7a1` [00:10:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-cb4c7a1 - 13https://git.io/vbL5b [00:10:53] [02mediawiki] 07dependabot[bot] opened pull request 03#1415: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `cb4c7a1` - 13https://git.io/JYSbC [00:10:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1415: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `cb4c7a1` - 13https://git.io/JYSbC [00:10:56] [02mediawiki] 07dependabot[bot] labeled pull request 03#1415: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `cb4c7a1` - 13https://git.io/JYSbC [00:10:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#1397: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `3af22f6` - 13https://git.io/JYSbW [00:10:59] [02mediawiki] 07dependabot[bot] closed pull request 03#1397: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `3af22f6` - 13https://git.io/JYcAu [00:11:00] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-3af22f6 - 13https://git.io/vbL5b [00:11:02] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-3af22f6 [00:11:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Math-f849125 [+0/-0/±1] 13https://git.io/JYSbl [00:11:05] [02miraheze/mediawiki] 07dependabot[bot] 03097f6b8 - Bump extensions/Math from `a412f37` to `f849125` [00:11:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Math-f849125 - 13https://git.io/vbL5b [00:11:08] [02mediawiki] 07dependabot[bot] opened pull request 03#1416: Bump extensions/Math from `a412f37` to `f849125` - 13https://git.io/JYSb8 [00:11:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1416: Bump extensions/Math from `a412f37` to `f849125` - 13https://git.io/JYSb8 [00:11:11] [02mediawiki] 07dependabot[bot] labeled pull request 03#1416: Bump extensions/Math from `a412f37` to `f849125` - 13https://git.io/JYSb8 [00:11:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/AbuseFilter-6eb8dd9 [+0/-0/±1] 13https://git.io/JYSb4 [00:11:14] [02miraheze/mediawiki] 07dependabot[bot] 032deae74 - Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` [00:11:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/AbuseFilter-6eb8dd9 - 13https://git.io/vbL5b [00:11:17] [02mediawiki] 07dependabot[bot] opened pull request 03#1417: Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` - 13https://git.io/JYSbB [00:11:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#1417: Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` - 13https://git.io/JYSbB [00:11:20] [02mediawiki] 07dependabot[bot] labeled pull request 03#1417: Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` - 13https://git.io/JYSbB [00:11:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Wikibase-0cd4d06 [+0/-0/±1] 13https://git.io/JYSbR [00:11:23] [02miraheze/mediawiki] 07dependabot[bot] 03bb67266 - Bump extensions/Wikibase from `292153f` to `0cd4d06` [00:11:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Wikibase-0cd4d06 - 13https://git.io/vbL5b [00:11:26] [02mediawiki] 07dependabot[bot] opened pull request 03#1418: Bump extensions/Wikibase from `292153f` to `0cd4d06` - 13https://git.io/JYSb0 [00:11:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#1418: Bump extensions/Wikibase from `292153f` to `0cd4d06` - 13https://git.io/JYSb0 [00:11:29] [02mediawiki] 07dependabot[bot] labeled pull request 03#1418: Bump extensions/Wikibase from `292153f` to `0cd4d06` - 13https://git.io/JYSb0 [00:11:30] [02mediawiki] 07dependabot[bot] commented on pull request 03#1396: Bump extensions/Wikibase from `292153f` to `38d1b49` - 13https://git.io/JYSbE [00:11:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/MatomoAnalytics-4b720ff [+0/-0/±1] 13https://git.io/JYSbu [00:11:33] [02miraheze/mediawiki] 07dependabot[bot] 039212f09 - Bump extensions/MatomoAnalytics from `aacdf8c` to `4b720ff` [00:11:35] [02mediawiki] 07dependabot[bot] closed pull request 03#1396: Bump extensions/Wikibase from `292153f` to `38d1b49` - 13https://git.io/JYcAR [00:11:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/MatomoAnalytics-4b720ff - 13https://git.io/vbL5b [00:11:38] [02mediawiki] 07dependabot[bot] opened pull request 03#1419: Bump extensions/MatomoAnalytics from `aacdf8c` to `4b720ff` - 13https://git.io/JYSbz [00:11:39] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Wikibase-38d1b49 - 13https://git.io/vbL5b [00:11:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Wikibase-38d1b49 [00:11:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#1419: Bump extensions/MatomoAnalytics from `aacdf8c` to `4b720ff` - 13https://git.io/JYSbz [00:11:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#1419: Bump extensions/MatomoAnalytics from `aacdf8c` to `4b720ff` - 13https://git.io/JYSbz [00:11:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Comments-28e2755 [+0/-0/±1] 13https://git.io/JYSbg [00:11:47] [02miraheze/mediawiki] 07dependabot[bot] 03fc426c7 - Bump extensions/Comments from `d75fa77` to `28e2755` [00:11:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Comments-28e2755 - 13https://git.io/vbL5b [00:11:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.34, 7.64, 7.19 [00:11:50] [02mediawiki] 07dependabot[bot] opened pull request 03#1420: Bump extensions/Comments from `d75fa77` to `28e2755` - 13https://git.io/JYSb2 [00:11:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#1420: Bump extensions/Comments from `d75fa77` to `28e2755` - 13https://git.io/JYSb2 [00:11:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1420: Bump extensions/Comments from `d75fa77` to `28e2755` - 13https://git.io/JYSb2 [00:11:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#1394: Bump extensions/Comments from `d75fa77` to `bc92e45` - 13https://git.io/JYSba [00:11:56] [02mediawiki] 07dependabot[bot] closed pull request 03#1394: Bump extensions/Comments from `d75fa77` to `bc92e45` - 13https://git.io/JYcAC [00:11:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-bc92e45 - 13https://git.io/vbL5b [00:11:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-bc92e45 [00:12:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Cosmos-e54190f [+0/-0/±1] 13https://git.io/JYSbw [00:12:02] [02miraheze/mediawiki] 07dependabot[bot] 032cc0558 - Bump skins/Cosmos from `e3b643f` to `e54190f` [00:12:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Cosmos-e54190f - 13https://git.io/vbL5b [00:12:05] [02mediawiki] 07dependabot[bot] opened pull request 03#1421: Bump skins/Cosmos from `e3b643f` to `e54190f` - 13https://git.io/JYSbr [00:12:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#1421: Bump skins/Cosmos from `e3b643f` to `e54190f` - 13https://git.io/JYSbr [00:12:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#1421: Bump skins/Cosmos from `e3b643f` to `e54190f` - 13https://git.io/JYSbr [00:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ProofreadPage-63a93d5 [+0/-0/±1] 13https://git.io/JYSbK [00:12:11] [02miraheze/mediawiki] 07dependabot[bot] 03b8a24c0 - Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` [00:12:12] [02mediawiki] 07dependabot[bot] opened pull request 03#1422: Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` - 13https://git.io/JYSb6 [00:12:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ProofreadPage-63a93d5 - 13https://git.io/vbL5b [00:12:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#1422: Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` - 13https://git.io/JYSb6 [00:12:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#1422: Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` - 13https://git.io/JYSb6 [00:12:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/VisualEditor-dc85118 [+0/-0/±1] 13https://git.io/JYSbi [00:12:20] [02miraheze/mediawiki] 07dependabot[bot] 03f51bac8 - Bump extensions/VisualEditor from `be34a2d` to `dc85118` [00:12:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/VisualEditor-dc85118 - 13https://git.io/vbL5b [00:12:23] [02mediawiki] 07dependabot[bot] opened pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYSbP [00:12:24] [02mediawiki] 07dependabot[bot] labeled pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYSbP [00:12:26] [02mediawiki] 07dependabot[bot] labeled pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYSbP [00:12:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Video-b7f2524 [+0/-0/±1] 13https://git.io/JYSbX [00:12:29] [02miraheze/mediawiki] 07dependabot[bot] 036e7798c - Bump extensions/Video from `8146fd8` to `b7f2524` [00:12:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Video-b7f2524 - 13https://git.io/vbL5b [00:12:32] [02mediawiki] 07dependabot[bot] opened pull request 03#1424: Bump extensions/Video from `8146fd8` to `b7f2524` - 13https://git.io/JYSb1 [00:12:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1424: Bump extensions/Video from `8146fd8` to `b7f2524` - 13https://git.io/JYSb1 [00:12:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#1424: Bump extensions/Video from `8146fd8` to `b7f2524` - 13https://git.io/JYSb1 [00:12:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/MinervaNeue-4f16566 [+0/-0/±1] 13https://git.io/JYSbD [00:12:38] [02miraheze/mediawiki] 07dependabot[bot] 03ace21a6 - Bump skins/MinervaNeue from `97729f6` to `4f16566` [00:12:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/MinervaNeue-4f16566 - 13https://git.io/vbL5b [00:12:41] [02mediawiki] 07dependabot[bot] opened pull request 03#1425: Bump skins/MinervaNeue from `97729f6` to `4f16566` - 13https://git.io/JYSby [00:12:42] [02mediawiki] 07dependabot[bot] labeled pull request 03#1425: Bump skins/MinervaNeue from `97729f6` to `4f16566` - 13https://git.io/JYSby [00:12:44] [02mediawiki] 07dependabot[bot] labeled pull request 03#1425: Bump skins/MinervaNeue from `97729f6` to `4f16566` - 13https://git.io/JYSby [00:12:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Translate-dee76bd [+0/-0/±1] 13https://git.io/JYSb9 [00:12:47] [02miraheze/mediawiki] 07dependabot[bot] 03149ddc5 - Bump extensions/Translate from `6d5bcca` to `dee76bd` [00:12:48] [02mediawiki] 07dependabot[bot] opened pull request 03#1426: Bump extensions/Translate from `6d5bcca` to `dee76bd` - 13https://git.io/JYSbH [00:12:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Translate-dee76bd - 13https://git.io/vbL5b [00:12:51] [02mediawiki] 07dependabot[bot] labeled pull request 03#1426: Bump extensions/Translate from `6d5bcca` to `dee76bd` - 13https://git.io/JYSbH [00:12:53] [02mediawiki] 07dependabot[bot] labeled pull request 03#1426: Bump extensions/Translate from `6d5bcca` to `dee76bd` - 13https://git.io/JYSbH [00:12:54] [02mediawiki] 07dependabot[bot] commented on pull request 03#1400: Bump extensions/Translate from `6d5bcca` to `05e2fdb` - 13https://git.io/JYSbQ [00:12:56] [02mediawiki] 07dependabot[bot] closed pull request 03#1400: Bump extensions/Translate from `6d5bcca` to `05e2fdb` - 13https://git.io/JYcAr [00:12:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Translate-05e2fdb - 13https://git.io/vbL5b [00:12:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Translate-05e2fdb [00:13:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/TemplateWizard-439da85 [+0/-0/±1] 13https://git.io/JYSb7 [00:13:02] [02miraheze/mediawiki] 07dependabot[bot] 03f5f9957 - Bump extensions/TemplateWizard from `9b8d627` to `439da85` [00:13:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/TemplateWizard-439da85 - 13https://git.io/vbL5b [00:13:05] [02mediawiki] 07dependabot[bot] opened pull request 03#1427: Bump extensions/TemplateWizard from `9b8d627` to `439da85` - 13https://git.io/JYSbd [00:13:06] [02mediawiki] 07dependabot[bot] labeled pull request 03#1427: Bump extensions/TemplateWizard from `9b8d627` to `439da85` - 13https://git.io/JYSbd [00:13:08] [02mediawiki] 07dependabot[bot] labeled pull request 03#1427: Bump extensions/TemplateWizard from `9b8d627` to `439da85` - 13https://git.io/JYSbd [00:13:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Vector-57326a0 [+0/-0/±1] 13https://git.io/JYSbF [00:13:11] [02miraheze/mediawiki] 07dependabot[bot] 03d4231d2 - Bump skins/Vector from `2ac57c8` to `57326a0` [00:13:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Vector-57326a0 - 13https://git.io/vbL5b [00:13:14] [02mediawiki] 07dependabot[bot] opened pull request 03#1428: Bump skins/Vector from `2ac57c8` to `57326a0` - 13https://git.io/JYSbb [00:13:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#1428: Bump skins/Vector from `2ac57c8` to `57326a0` - 13https://git.io/JYSbb [00:13:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#1428: Bump skins/Vector from `2ac57c8` to `57326a0` - 13https://git.io/JYSbb [00:13:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#1388: Bump skins/Vector from `2ac57c8` to `a57b2b6` - 13https://git.io/JYSbN [00:13:20] [02mediawiki] 07dependabot[bot] closed pull request 03#1388: Bump skins/Vector from `2ac57c8` to `a57b2b6` - 13https://git.io/JYcAU [00:13:21] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Vector-a57b2b6 - 13https://git.io/vbL5b [00:13:23] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Vector-a57b2b6 [00:13:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/OAuth-b697ceb [+0/-0/±1] 13https://git.io/JYSbA [00:13:26] [02miraheze/mediawiki] 07dependabot[bot] 036e5734a - Bump extensions/OAuth from `5a56193` to `b697ceb` [00:13:27] [02mediawiki] 07dependabot[bot] opened pull request 03#1429: Bump extensions/OAuth from `5a56193` to `b697ceb` - 13https://git.io/JYSbx [00:13:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/OAuth-b697ceb - 13https://git.io/vbL5b [00:13:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#1429: Bump extensions/OAuth from `5a56193` to `b697ceb` - 13https://git.io/JYSbx [00:13:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#1429: Bump extensions/OAuth from `5a56193` to `b697ceb` - 13https://git.io/JYSbx [00:13:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Newsletter-1b6a18c [+0/-0/±1] 13https://git.io/JYSbh [00:13:35] [02miraheze/mediawiki] 07dependabot[bot] 03557e7db - Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` [00:13:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Newsletter-1b6a18c - 13https://git.io/vbL5b [00:13:38] [02mediawiki] 07dependabot[bot] opened pull request 03#1430: Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` - 13https://git.io/JYSbj [00:13:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#1430: Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` - 13https://git.io/JYSbj [00:13:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#1430: Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` - 13https://git.io/JYSbj [00:13:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PortableInfobox-f713ac0 [+0/-0/±1] 13https://git.io/JYSNe [00:13:44] [02miraheze/mediawiki] 07dependabot[bot] 031373cad - Bump extensions/PortableInfobox from `7efce76` to `f713ac0` [00:13:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PortableInfobox-f713ac0 - 13https://git.io/vbL5b [00:13:47] [02mediawiki] 07dependabot[bot] opened pull request 03#1431: Bump extensions/PortableInfobox from `7efce76` to `f713ac0` - 13https://git.io/JYSNv [00:13:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.10, 7.46, 7.18 [00:13:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#1431: Bump extensions/PortableInfobox from `7efce76` to `f713ac0` - 13https://git.io/JYSNv [00:13:50] [02mediawiki] 07dependabot[bot] labeled pull request 03#1431: Bump extensions/PortableInfobox from `7efce76` to `f713ac0` - 13https://git.io/JYSNv [00:13:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PollNY-bc8e107 [+0/-0/±1] 13https://git.io/JYSNf [00:13:53] [02miraheze/mediawiki] 07dependabot[bot] 03c055257 - Bump extensions/PollNY from `edbf58e` to `bc8e107` [00:13:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PollNY-bc8e107 - 13https://git.io/vbL5b [00:13:56] [02mediawiki] 07dependabot[bot] opened pull request 03#1432: Bump extensions/PollNY from `edbf58e` to `bc8e107` - 13https://git.io/JYSNJ [00:13:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#1432: Bump extensions/PollNY from `edbf58e` to `bc8e107` - 13https://git.io/JYSNJ [00:13:59] [02mediawiki] 07dependabot[bot] labeled pull request 03#1432: Bump extensions/PollNY from `edbf58e` to `bc8e107` - 13https://git.io/JYSNJ [00:14:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#1383: Bump extensions/PollNY from `edbf58e` to `f18573d` - 13https://git.io/JYSNU [00:14:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CreateWiki-e60c4a5 [+0/-0/±1] 13https://git.io/JYSNT [00:14:03] [02miraheze/mediawiki] 07dependabot[bot] 036b5769c - Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` [00:14:05] [02mediawiki] 07dependabot[bot] opened pull request 03#1433: Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` - 13https://git.io/JYSNk [00:14:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.75, 6.52, 6.62 [00:14:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CreateWiki-e60c4a5 - 13https://git.io/vbL5b [00:14:08] [02mediawiki] 07dependabot[bot] closed pull request 03#1383: Bump extensions/PollNY from `edbf58e` to `f18573d` - 13https://git.io/JYcN7 [00:14:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1433: Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` - 13https://git.io/JYSNk [00:14:11] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/PollNY-f18573d [00:14:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#1433: Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` - 13https://git.io/JYSNk [00:14:14] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/PollNY-f18573d - 13https://git.io/vbL5b [00:14:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CommentStreams-ed3fc73 [+0/-0/±1] 13https://git.io/JYSNL [00:14:17] [02miraheze/mediawiki] 07dependabot[bot] 03a6a70c2 - Bump extensions/CommentStreams from `bbae251` to `ed3fc73` [00:14:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-ed3fc73 - 13https://git.io/vbL5b [00:14:20] [02mediawiki] 07dependabot[bot] opened pull request 03#1434: Bump extensions/CommentStreams from `bbae251` to `ed3fc73` - 13https://git.io/JYSNt [00:14:21] [02mediawiki] 07dependabot[bot] labeled pull request 03#1434: Bump extensions/CommentStreams from `bbae251` to `ed3fc73` - 13https://git.io/JYSNt [00:14:23] [02mediawiki] 07dependabot[bot] labeled pull request 03#1434: Bump extensions/CommentStreams from `bbae251` to `ed3fc73` - 13https://git.io/JYSNt [00:14:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/WikiDiscover-dc538a5 [+0/-0/±1] 13https://git.io/JYSNq [00:14:26] [02miraheze/mediawiki] 07dependabot[bot] 03230d21e - Bump extensions/WikiDiscover from `d69574d` to `dc538a5` [00:14:27] [02mediawiki] 07dependabot[bot] opened pull request 03#1435: Bump extensions/WikiDiscover from `d69574d` to `dc538a5` - 13https://git.io/JYSNm [00:14:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/WikiDiscover-dc538a5 - 13https://git.io/vbL5b [00:14:30] [02mediawiki] 07dependabot[bot] labeled pull request 03#1435: Bump extensions/WikiDiscover from `d69574d` to `dc538a5` - 13https://git.io/JYSNm [00:14:32] [02mediawiki] 07dependabot[bot] labeled pull request 03#1435: Bump extensions/WikiDiscover from `d69574d` to `dc538a5` - 13https://git.io/JYSNm [00:14:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-5668e4e [+0/-0/±1] 13https://git.io/JYSNY [00:14:35] [02miraheze/mediawiki] 07dependabot[bot] 034e01687 - Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` [00:14:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-5668e4e - 13https://git.io/vbL5b [00:14:38] [02mediawiki] 07dependabot[bot] opened pull request 03#1436: Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` - 13https://git.io/JYSNO [00:14:39] [02mediawiki] 07dependabot[bot] labeled pull request 03#1436: Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` - 13https://git.io/JYSNO [00:14:41] [02mediawiki] 07dependabot[bot] labeled pull request 03#1436: Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` - 13https://git.io/JYSNO [00:14:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Echo-17c90bc [+0/-0/±1] 13https://git.io/JYSN3 [00:14:44] [02miraheze/mediawiki] 07dependabot[bot] 03e1ee4ac - Bump extensions/Echo from `fd6a33e` to `17c90bc` [00:14:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Echo-17c90bc - 13https://git.io/vbL5b [00:14:47] [02mediawiki] 07dependabot[bot] opened pull request 03#1437: Bump extensions/Echo from `fd6a33e` to `17c90bc` - 13https://git.io/JYSNs [00:14:48] [02mediawiki] 07dependabot[bot] labeled pull request 03#1437: Bump extensions/Echo from `fd6a33e` to `17c90bc` - 13https://git.io/JYSNs [00:14:50] [02mediawiki] 07dependabot[bot] labeled pull request 03#1437: Bump extensions/Echo from `fd6a33e` to `17c90bc` - 13https://git.io/JYSNs [00:14:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/QuizGame-a7c82f2 [+0/-0/±1] 13https://git.io/JYSNn [00:14:53] [02miraheze/mediawiki] 07dependabot[bot] 033d3baaa - Bump extensions/QuizGame from `1a9fbc3` to `a7c82f2` [00:14:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/QuizGame-a7c82f2 - 13https://git.io/vbL5b [00:14:56] [02mediawiki] 07dependabot[bot] opened pull request 03#1438: Bump extensions/QuizGame from `1a9fbc3` to `a7c82f2` - 13https://git.io/JYSNc [00:14:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#1438: Bump extensions/QuizGame from `1a9fbc3` to `a7c82f2` - 13https://git.io/JYSNc [00:14:59] [02mediawiki] 07dependabot[bot] labeled pull request 03#1438: Bump extensions/QuizGame from `1a9fbc3` to `a7c82f2` - 13https://git.io/JYSNc [00:15:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#1379: Bump extensions/QuizGame from `1a9fbc3` to `efea676` - 13https://git.io/JYSNC [00:15:02] [02mediawiki] 07dependabot[bot] closed pull request 03#1379: Bump extensions/QuizGame from `1a9fbc3` to `efea676` - 13https://git.io/JYcN6 [00:15:03] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/QuizGame-efea676 [00:15:05] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/QuizGame-efea676 - 13https://git.io/vbL5b [00:15:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/SocialProfile-42c0593 [+0/-0/±1] 13https://git.io/JYSNW [00:15:08] [02miraheze/mediawiki] 07dependabot[bot] 03c7a5dd6 - Bump extensions/SocialProfile from `fbb875b` to `42c0593` [00:15:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-42c0593 - 13https://git.io/vbL5b [00:15:11] [02mediawiki] 07dependabot[bot] opened pull request 03#1439: Bump extensions/SocialProfile from `fbb875b` to `42c0593` - 13https://git.io/JYSN8 [00:15:12] [02mediawiki] 07dependabot[bot] labeled pull request 03#1439: Bump extensions/SocialProfile from `fbb875b` to `42c0593` - 13https://git.io/JYSN8 [00:15:14] [02mediawiki] 07dependabot[bot] labeled pull request 03#1439: Bump extensions/SocialProfile from `fbb875b` to `42c0593` - 13https://git.io/JYSN8 [00:15:15] [02mediawiki] 07dependabot[bot] commented on pull request 03#1390: Bump extensions/SocialProfile from `fbb875b` to `00cd0f1` - 13https://git.io/JYSN4 [00:15:17] [02mediawiki] 07dependabot[bot] closed pull request 03#1390: Bump extensions/SocialProfile from `fbb875b` to `00cd0f1` - 13https://git.io/JYcAq [00:15:18] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-00cd0f1 [00:15:20] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-00cd0f1 - 13https://git.io/vbL5b [00:15:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ContributionScores-20b9a56 [+0/-0/±1] 13https://git.io/JYSNB [00:15:23] [02miraheze/mediawiki] 07dependabot[bot] 035313b76 - Bump extensions/ContributionScores from `6fac009` to `20b9a56` [00:15:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ContributionScores-20b9a56 - 13https://git.io/vbL5b [00:15:26] [02mediawiki] 07dependabot[bot] opened pull request 03#1440: Bump extensions/ContributionScores from `6fac009` to `20b9a56` - 13https://git.io/JYSNR [00:15:27] [02mediawiki] 07dependabot[bot] labeled pull request 03#1440: Bump extensions/ContributionScores from `6fac009` to `20b9a56` - 13https://git.io/JYSNR [00:15:29] [02mediawiki] 07dependabot[bot] labeled pull request 03#1440: Bump extensions/ContributionScores from `6fac009` to `20b9a56` - 13https://git.io/JYSNR [00:15:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/VoteNY-fe23523 [+0/-0/±1] 13https://git.io/JYSN0 [00:15:32] [02miraheze/mediawiki] 07dependabot[bot] 0365dfc7c - Bump extensions/VoteNY from `c7a829a` to `fe23523` [00:15:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/VoteNY-fe23523 - 13https://git.io/vbL5b [00:15:35] [02mediawiki] 07dependabot[bot] opened pull request 03#1441: Bump extensions/VoteNY from `c7a829a` to `fe23523` - 13https://git.io/JYSNE [00:15:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#1441: Bump extensions/VoteNY from `c7a829a` to `fe23523` - 13https://git.io/JYSNE [00:15:38] [02mediawiki] 07dependabot[bot] labeled pull request 03#1441: Bump extensions/VoteNY from `c7a829a` to `fe23523` - 13https://git.io/JYSNE [00:15:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.77, 6.27, 6.77 [00:25:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.29, 6.28, 5.85 [00:27:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.29, 6.07, 5.84 [00:35:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.25, 7.56, 6.43 [00:35:36] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.11, 7.50, 6.42 [00:37:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.24, 7.36, 6.49 [00:37:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.99, 7.43, 6.55 [00:40:54] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.35, 6.53, 6.37 [00:41:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.51, 6.07, 6.21 [00:56:54] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [01:04:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 6.17, 5.40 [01:06:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.92, 5.54, 5.25 [01:20:39] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Diff, 31032 files, 38.84GB, 2021-03-21 01:17:00 (2.1 weeks ago) [02:04:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.65, 8.77, 6.18 [02:06:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.67, 7.17, 5.91 [02:08:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.51, 5.85, 5.57 [03:55:31] [02mw-config] 07joritochip opened pull request 03#3814: Add simulatorwiki configs (T7094) - 13https://git.io/JY9Wq [03:56:35] miraheze/mw-config - joritochip the build passed. [03:57:05] [02mw-config] 07joritochip edited pull request 03#3814: Add simulatorwiki configs (T7094) - 13https://git.io/JY9Wq [04:00:54] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [05:13:11] [02mw-config] 07Reception123 closed pull request 03#3814: Add simulatorwiki configs (T7094) - 13https://git.io/JY9Wq [05:13:13] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JY9Vl [05:13:14] [02miraheze/mw-config] 07joritochip 0311fa653 - Add simulatorwiki configs (T7094) (#3814) [05:13:29] [02mw-config] 07Reception123 commented on pull request 03#3814: Add simulatorwiki configs (T7094) - 13https://git.io/JY9VB [05:14:09] miraheze/mw-config - Reception123 the build passed. [05:47:35] !log reception@jobrunner3:/srv/mediawiki/w/extensions/AJAXPoll/maintenance$ sudo -u www-data php migrateOldAJAXPollUserColumnsToActor.php --wiki=rottenwebsiteswiki [05:47:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:48:30] !log MariaDB [rottenwebsiteswiki]> ALTER TABLE /*_*/ajaxpoll_vote ADD COLUMN poll_actor bigint unsigned NOT NULL AFTER poll_id; (before the script) [05:48:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:28] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.53, 4.87, 2.46 [07:49:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.20, 3.34, 2.19 [10:31:29] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.55, 3.57, 2.29 [10:33:29] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.65, 2.56, 2.07 [10:56:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.42, 6.43, 5.42 [10:58:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.13, 5.52, 5.21 [12:02:36] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.30, 3.70, 2.02 [12:04:36] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.05, 2.66, 1.84 [12:04:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.60, 5.97, 4.91 [12:06:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.71, 6.05, 5.06 [12:11:28] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.42, 4.52, 2.89 [12:13:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.70, 3.10, 2.56 [12:32:44] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.85, 3.49, 1.92 [12:34:44] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.58, 2.47, 1.74 [12:37:28] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.53, 5.25, 3.63 [12:38:44] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.02, 3.87, 2.58 [12:39:30] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.45, 4.00, 3.37 [12:40:44] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.48, 2.97, 2.40 [12:41:29] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.51, 2.81, 3.00 [13:03:28] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.99, 4.18, 2.90 [13:05:28] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.80, 3.59, 2.81 [13:07:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.53, 2.74, 2.59 [13:18:05] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 432.80 ms [13:23:28] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.67, 3.24, 2.44 [13:25:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.81, 2.37, 2.22 [13:26:04] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 257.76 ms [13:31:37] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-1/±2] 13https://git.io/JYHH1 [13:31:39] [02miraheze/MirahezeMagic] 07translatewiki 039b10663 - Localisation updates from https://translatewiki.net. [13:31:40] [ Main page - translatewiki.net ] - translatewiki.net [13:32:40] miraheze/MirahezeMagic - translatewiki the build passed. [13:44:10] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr* [13:44:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:47:18] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.92, 4.44, 3.00 [13:48:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.20, 6.58, 4.95 [13:48:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.25, 6.13, 4.85 [13:48:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.03, 7.05, 5.18 [13:49:17] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.04, 3.59, 2.88 [13:49:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.63, 7.31, 5.20 [13:51:18] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.32, 2.76, 2.65 [13:52:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.80, 7.77, 5.86 [13:52:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.23, 7.82, 5.98 [13:54:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.96, 8.40, 6.31 [13:54:34] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.08, 1.67, 1.03 [13:54:46] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.33, 8.43, 6.43 [13:56:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.80, 7.95, 6.36 [13:56:31] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.70, 1.25, 0.95 [13:58:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.67, 8.48, 6.76 [14:06:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.08, 7.75, 7.19 [14:06:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.50, 6.96, 7.03 [14:08:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.37, 7.51, 7.31 [14:08:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.73, 7.20, 7.16 [14:10:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.75, 6.26, 6.71 [14:12:47] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.56, 5.91, 6.60 [14:14:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.29, 5.85, 6.60 [14:14:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.12, 5.92, 6.66 [14:21:28] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYHxP [14:21:30] [02miraheze/puppet] 07Southparkfan 03c32785b - Varnish: block Googlebot requests with specific parameter [14:22:20] $log ban requests re puppet@c32785b via search console [14:22:24] !log ban requests re puppet@c32785b via search console [14:22:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:23:09] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.99, 3.87, 2.73 [14:25:07] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.83, 2.81, 2.47 [14:26:19] SPF|Cloud https://github.com/miraheze/puppet/blob/master/modules/varnish/templates/default.vcl#L405 wouldn't that mean it won't cache if the url is /wiki/ or index.php? [14:26:20] [ puppet/default.vcl at master · miraheze/puppet · GitHub ] - github.com [14:27:08] paladox: it tells the client not to cache response [14:27:45] vcl_deliver affects the response to the client (the reader/editor), not the backend response (mediawiki -> varnish) [14:27:45] i don't think that's something we wont? At least comparing en.wikipedia.org to meta shows a different cache. [14:28:15] e.g i see "cache-control: s-maxage=600" for en.w.org and for meta "cache-control: private, s-maxage=0, maxage=0, must-revalidate" [14:29:07] hmm [14:29:09] paladox: s-maxage is used by caches, such as proxies [14:30:05] why does en.w.org have s-maxage=600 and where is that configured [14:31:00] certain companies have proxies for their employees (to monitor employee traffic, for example), the en.wikipedia.org could be useful to add extra caching to that proxy layer [14:31:24] other than that I see no reason to add s-maxage=600 [14:32:03] and regarding your questions, no answer [14:32:08] ah ok [14:32:12] but I do not see a reason to do the same here [14:34:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.29, 6.39, 6.04 [14:40:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.06, 6.60, 6.31 [14:53:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQJK [14:53:30] [02miraheze/puppet] 07paladox 03a587f53 - varnish: Minor improvements [14:53:32] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [14:53:33] [02puppet] 07paladox opened pull request 03#1727: varnish: Minor improvements - 13https://git.io/JYQJi [14:57:37] [02mw-config] 07joritochip opened pull request 03#3815: Configure LinkTitles for simulatorwiki per T7094 - 13https://git.io/JYQUX [14:57:56] [02mw-config] 07joritochip edited pull request 03#3815: Configure LinkTitles for simulatorwiki per T7094 - 13https://git.io/JYQUX [14:58:47] miraheze/mw-config - joritochip the build passed. [15:00:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQT3 [15:00:55] [02miraheze/puppet] 07paladox 033f19812 - Update default.vcl [15:00:57] [02puppet] 07paladox synchronize pull request 03#1727: varnish: Minor improvements - 13https://git.io/JYQJi [15:01:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQTC [15:01:41] [02miraheze/puppet] 07paladox 03d2ffd73 - Update default.vcl [15:01:43] [02puppet] 07paladox synchronize pull request 03#1727: varnish: Minor improvements - 13https://git.io/JYQJi [15:02:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQTu [15:02:26] [02miraheze/puppet] 07paladox 039e42fd7 - Update default.vcl [15:02:28] [02puppet] 07paladox synchronize pull request 03#1727: varnish: Minor improvements - 13https://git.io/JYQJi [15:07:50] [02mw-config] 07paladox closed pull request 03#3815: Configure LinkTitles for simulatorwiki per T7094 - 13https://git.io/JYQUX [15:07:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQkK [15:07:53] [02miraheze/mw-config] 07joritochip 03c22e019 - Configure LinkTitles for simulatorwiki per T7094 (#3815) [15:08:50] miraheze/mw-config - paladox the build passed. [15:19:45] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQtr [15:19:46] [02miraheze/puppet] 07Southparkfan 03bb1ec90 - Revert "Varnish: block Googlebot requests with specific parameter" [15:24:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.37, 6.16, 5.17 [15:24:43] !log deleted and dropped simpleelectronicswiki (reset) [15:24:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.77, 8.31, 5.98 [15:26:08] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.69, 5.79, 5.15 [15:26:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.90, 6.24, 5.24 [15:28:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.77, 6.21, 5.34 [15:30:41] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki simpleelectronicswiki /home/reception/resets/simpleelectronicswiki.xml username-prefix="simpleelectronics" [15:30:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:31:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 7.43, 6.42 [15:33:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.23, 7.90, 6.67 [15:33:29] !log reception@jobrunner3:~/resets$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki simpleelectronicswiki /home/reception/resets/mnt --search-recursively [15:33:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.87, 8.74, 6.75 [15:33:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.01, 6.52, 5.75 [15:35:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.78, 7.93, 6.31 [15:35:17] !log MariaDB [simpleelectronicswiki]> INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); [15:35:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:08] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQOA [15:36:10] [02miraheze/puppet] 07Southparkfan 030b1a5da - Revert "Revert "Varnish: block Googlebot requests with specific parameter"" [15:36:55] [02puppet] 07paladox closed pull request 03#1727: varnish: Minor improvements - 13https://git.io/JYQJi [15:36:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQ3O [15:36:58] [02miraheze/puppet] 07paladox 03baeffde - varnish: Minor improvements (#1727) [15:36:59] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [15:37:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [15:37:01] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.27, 7.52, 6.20 [15:37:20] load issues may be solved soon [15:37:23] on mediawiki servers [15:38:24] and my explanation for the increase in backend requests after varnish patches: restarts flush the cache :) [15:40:39] !sre Just got an error 502 Bad gateway trying to view preview on Meta [15:40:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 7.84, 6.68 [15:41:13] uh [15:41:15] let me revert [15:41:22] have no issues here [15:41:28] Resolved now. [15:41:31] R4356th: which cache proxy are you using? [15:41:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQsR [15:41:59] [02miraheze/puppet] 07paladox 030388982 - Revert "varnish: Minor improvements (#1727)" [15:42:29] SPF|Cloud: cp3 [15:42:33] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[varnish-server-syntax] [15:42:35] let's see [15:43:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±1] 13https://git.io/JYQGe [15:43:52] [02miraheze/puppet] 07paladox 030aaefcf - Revert "Revert "varnish: Minor improvements (#1727)"" [15:44:09] oh [15:44:10] bugger [15:44:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-2/±1] 13https://git.io/JYQGs [15:44:31] [02miraheze/puppet] 07paladox 037f74ed2 - Revert "Revert "Revert "varnish: Minor improvements (#1727)""" [15:44:34] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:44:58] what are you doing paladox? [15:45:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.42, 7.48, 7.16 [15:45:30] SPF|Cloud i accidentally included some changes i didn't want commited & my original change had a regex that was invalid so i reverted it. [15:45:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQGo [15:45:55] [02miraheze/puppet] 07paladox 0370dcc91 - varnish: Minor improvements [15:45:56] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [15:45:58] [02puppet] 07paladox opened pull request 03#1728: varnish: Minor improvements - 13https://git.io/JYQGK [15:46:06] SPF|Cloud should ^ work [15:46:09] fixed the regex [15:46:18] good [15:46:43] R4356th: cannot find an error in varnish logs [15:46:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.65, 7.96, 7.55 [15:47:02] 🤔 [15:47:10] can you PM me the first octets of your IP? [15:47:27] first two are sufficient [15:47:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.54, 7.38, 7.61 [15:47:35] Not on IRC. (Banned in error by Freenode) [15:47:40] discord is fine [15:47:41] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[varnish-server-syntax] [15:47:49] Okay then. [15:48:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.12, 6.23, 6.37 [15:49:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.54, 6.11, 6.71 [15:49:14] likely cause of your error is the varnish restart [15:50:11] if you see SREs performing maintenance here, they can be expected, otherwise 502s are not normal for sure [15:50:38] Oh, I see. Thank you. (I was not following this channel when I got the error.) [15:50:48] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.18, 5.67, 6.73 [15:57:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 2.96, 5.09, 6.46 [16:08:47] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.44, 6.45, 6.11 [16:10:17] !log re-deleted simpleelectronicswiki [16:10:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.04, 6.27, 6.09 [16:15:40] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:25:48] !log renamed testdelete2wiki to simpleelectronicswiki [16:25:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:07] paladox: hmm, I wonder why the --logs prefix from dumpBackup.php doesn't actually work [16:27:52] hint: '-7849) AS nMkr WHERE 1216=1216 UNION ALL SELECT 24,24,24,24,24,24#' is not a normal user agent :) [16:29:59] SPF|Cloud: yeah [16:30:04] Will eat then chat [16:34:46] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.70, 5.74, 5.01 [16:36:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.16, 5.62, 5.04 [16:46:36] SPF|Cloud: what IPs? [16:58:10] No idea [17:11:52] SPF|Cloud: I looked, linux UA I've seen before and I've seen same country but bots doesn't keep logs long enough for firewall [17:12:05] From when someone pentested us [17:19:59] @Owen: see your email [17:28:11] !log (forgot to log before) but repeated imports logged above on testdelete2wiki (in order to rename afterwards) [17:28:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:37] paladox, if you're around, can you investigate `[5c89fb19f251b4f29d2f5c62] 2021-04-05 17:32:22: Fatal exception of type "TypeError"` on `simpleelectronicswiki`? Reception123 has run the trace, and it seems to be an OAuth issue that is preventing he and I from signing in manually to that recreated wiki. CentralAuth won't auto-log us in either [17:34:58] What is said trace dmehus [17:35:04] Let me get it [17:35:35] RhinosF1: ```Argument 1 passed to MediaWiki\Extension\OATHAuth\Auth\SecondaryAuthenticationProvider::getProviderForModule() must be an instance of MediaWiki\Extension\OATHAuth\IModule, null given, called in /srv/mediawiki/w/extensions/OATHAuth/src/Auth/SecondaryAuthenticationProvider.php on line``` [17:36:12] I see your ping but no trace [17:36:22] oh [17:36:25] let me repaste [17:36:36] ```Argument 1 passed to MediaWiki\Extension\OATHAuth\Auth\SecondaryAuthenticationProvider::getProviderForModule() must be an instance of MediaWiki\Extension\OATHAuth\IModule, null given, called in /srv/mediawiki/w/extensions/OATHAuth/src/Auth/SecondaryAuthenticationProvider.php on line``` [17:36:58] That's not a trace [17:37:07] The full thing please Reception123 [17:37:09] I've tried deleting my `simpleelectronics.miraheze.org` cookies several times, re-signing in to both Meta and that wiki several times, both before and after I've started my external authentication program (Authy) [17:37:13] RhinosF1, oh [17:37:34] Reception123, ^ [17:45:28] RhinosF1, do you have your laptop booted up? Maybe you could trace for him as he seems to be otherwise occupied [17:48:59] No I don't [17:49:03] Give me 10 [17:49:06] File a task [17:55:18] RhinosF1, I don't think we need a task for this, especially if it's upstream. It can wait a few minutes whoever provides full trace first, you or Reception123 [18:01:12] dmehus: if it's blocking login its task worthy [18:09:31] RhinosF1, ack okay, fair enough and we do have a steward and SRE who can't access that wiki logged in [18:09:40] Priority High maybe? [18:09:43] Yep [18:09:46] ok [18:09:48] doing now [18:11:42] https://graylog.miraheze.org/streams/5f78ae5246640840f1ef3d60/search?q=mediawiki_wiki%3Asimpleelectronicswiki&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=172800 [18:11:54] https://www.irccloud.com/pastebin/SWbo0Tfz/ [18:11:55] [ Snippet | IRCCloud ] - www.irccloud.com [18:15:21] RhinosF1, thanks...https://phabricator.miraheze.org/T7096 [18:15:22] [ ⚓ T7096 OAuth error on recreated simpleelectronicswiki ] - phabricator.miraheze.org [18:15:46] Ty [18:17:25] np [18:44:00] * Reception123 is back [18:47:18] wb [19:01:18] [02puppet] 07paladox closed pull request 03#1728: varnish: Minor improvements - 13https://git.io/JYQGK [19:01:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQ7N [19:01:21] [02miraheze/puppet] 07paladox 030a06652 - varnish: Minor improvements (#1728) [19:01:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [19:01:24] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:04:04] !sre I'm getting 502 bad gateways on `crappygameswiki` [19:04:27] Seems to be resolved though [19:04:44] Are we needed then? [19:05:20] I don't think so, not unless you want to refer to the Varnish access logs and see if other users encountered the same problem around that time? [19:05:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 5.73, 4.74 [19:05:35] I don't have varnish access logs [19:05:41] yeah, true [19:05:49] You'd see icinga-miraheze moaning if others got it [19:05:55] 502 -> mw [19:05:58] 503 -> cp [19:06:10] JohnLewis, ack, thanks :) [19:06:15] yeh it was a varnish restart that caused the 503 [19:06:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.12, 5.96, 4.92 [19:06:24] paladox: it's a 502 [19:06:29] Yeah that's a point 502 would come from mw [19:06:34] paladox, ack, okay, yeah just a temporary thing then I guess? [19:06:35] oh [19:06:40] yeh [19:06:57] paladox: could we depool when restarting varnish if it causes user impact? [19:07:22] But I don't see icinga-miraheze complaining so it can't have been long [19:07:24] we could. Requires doing it in the dns. [19:07:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.00, 5.12, 4.64 [19:07:45] paladox: is there not a way to apply config changes without breaking requests [19:08:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.45, 5.75, 4.95 [19:09:57] As far as i'm aware no. [19:10:19] :( [19:10:35] That's twice merging puppet changes affecting varnish has been noticed today [19:11:42] oh [19:11:46] actually [19:12:02] RhinosF1: well I mean it is kind of expected. Varnish/Nginx is basically our web server infrastructure, so restarting it would have impacts [19:12:06] this https://github.com/wikimedia/puppet/blob/production/modules/varnish/manifests/instance.pp#L193 [19:12:06] [ puppet/instance.pp at production · wikimedia/puppet · GitHub ] - github.com [19:12:26] at least the impacts are quite brief [19:12:48] dmehus: it's not expected. Stuff like that should have safe restart methods. [19:13:15] paladox: can you set that up? [19:13:43] RhinosF1, perhaps, yeah. Or perhaps we should have a custom Nginx error message that explains Varnish is being restarted [19:13:45] it's a chance that won't cause impact but i cannot gurentee tho. But yet [19:13:47] *yes [19:14:53] `Class['varnish::common::errorpage'],` suggests there might a custom error page we could set up for Varnish for these sort of things [19:15:17] We do have a custom error page already [19:15:35] ^ [19:15:59] dmehus: that's hard to detect why varnish is down or it wouldn't be down. It's more varnish needs an improved setup. I'm surprised it's long enough to cause impact tbh but a few systems have reload commands rather than restart or something [19:16:06] JohnLewis, yeah, can we maybe add to that and suggest some probable/possible causes for the error, and make it pretty (with a Miraheze logo)? [19:16:29] The Miraheze logo is already on the 503 error page [19:16:37] I meant the 502 page [19:16:39] but yeah [19:16:40] with possible causes, advice and so on [19:16:45] Improve the 502 page [19:16:45] the 502 isn't Varnish [19:16:49] oh [19:17:25] 502 is nginx [19:17:42] yeah [19:18:29] * dmehus gets confused on whether a 502 is a Varnish or Nginx issue. It's good to know it's not the former now, though [19:19:23] nginx doesn't have our custom error template [19:21:33] dmehus: 502 is nginx. 502 comes before a 503. [19:26:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQbU [19:26:21] [02miraheze/puppet] 07paladox 0373f83d0 - varnish: Use load-new-vcl-file exec [19:26:23] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:26:24] [02puppet] 07paladox opened pull request 03#1729: varnish: Use load-new-vcl-file exec - 13https://git.io/JYQbT [19:27:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQbq [19:27:40] [02miraheze/puppet] 07paladox 036018e3a - Update varnish.systemd.erb [19:27:42] [02puppet] 07paladox synchronize pull request 03#1729: varnish: Use load-new-vcl-file exec - 13https://git.io/JYQbT [19:30:13] [02puppet] 07paladox closed pull request 03#1729: varnish: Use load-new-vcl-file exec - 13https://git.io/JYQbT [19:30:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JYQbB [19:30:16] [02miraheze/puppet] 07paladox 032d21a1f - varnish: Use load-new-vcl-file exec (#1729) [19:30:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [19:30:19] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:31:20] paladox, and RhinosF1, ack [19:34:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.39, 6.86, 6.07 [19:36:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.15, 6.99, 6.23 [19:37:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQN6 [19:37:45] [02miraheze/puppet] 07paladox 039e8282e - varnish: Move test3 under the testing backend banner [19:37:47] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:37:48] [02puppet] 07paladox opened pull request 03#1730: varnish: Move test3 under the testing backend banner - 13https://git.io/JYQNi [19:37:54] [02puppet] 07paladox closed pull request 03#1730: varnish: Move test3 under the testing backend banner - 13https://git.io/JYQNi [19:37:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQNP [19:37:57] [02miraheze/puppet] 07paladox 0383f5943 - varnish: Move test3 under the testing backend banner (#1730) [19:37:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [19:38:00] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:38:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.56, 6.38, 6.09 [19:41:02] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [19:41:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQAm [19:41:25] [02miraheze/puppet] 07paladox 03c57e0bc - Fix [19:42:15] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [19:42:33] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [19:43:01] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:43:40] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [19:44:14] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:44:33] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:45:39] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:50:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQxR [19:50:23] [02miraheze/puppet] 07paladox 0306c3945 - varnish: Move some nginx related bits to varnish::nginx [19:50:24] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:50:26] [02puppet] 07paladox opened pull request 03#1731: varnish: Move some nginx related bits to varnish::nginx - 13https://git.io/JYQxE [19:53:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQxN [19:53:38] [02miraheze/puppet] 07paladox 031ad7e09 - Update nginx.pp [19:53:40] [02puppet] 07paladox synchronize pull request 03#1731: varnish: Move some nginx related bits to varnish::nginx - 13https://git.io/JYQxE [19:54:04] [02puppet] 07paladox closed pull request 03#1731: varnish: Move some nginx related bits to varnish::nginx - 13https://git.io/JYQxE [19:54:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JYQpe [19:54:07] [02miraheze/puppet] 07paladox 039b98276 - varnish: Move some nginx related bits to varnish::nginx (#1731) [19:54:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [19:54:10] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:56:11] seems to work - no 502 anymore [20:01:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQhI [20:01:42] [02miraheze/puppet] 07paladox 03e27cac9 - varnish: Move stunnel configuration to its own class [20:01:44] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [20:01:45] [02puppet] 07paladox opened pull request 03#1732: varnish: Move stunnel configuration to its own class - 13https://git.io/JYQhL [20:04:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JYQha [20:04:37] [02miraheze/puppet] 07paladox 035a2f1aa - Create stunnel4.pp [20:04:38] [02puppet] 07paladox synchronize pull request 03#1732: varnish: Move stunnel configuration to its own class - 13https://git.io/JYQhL [20:05:16] paladox, cool [20:05:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQhD [20:05:44] [02miraheze/puppet] 07paladox 03d263944 - Update stunnel4.pp [20:05:45] [02puppet] 07paladox synchronize pull request 03#1732: varnish: Move stunnel configuration to its own class - 13https://git.io/JYQhL [20:05:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQhS [20:05:51] [02miraheze/puppet] 07paladox 03effcae8 - Update init.pp [20:05:52] [02puppet] 07paladox synchronize pull request 03#1732: varnish: Move stunnel configuration to its own class - 13https://git.io/JYQhL [20:06:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYQh7 [20:06:18] [02miraheze/puppet] 07paladox 033758d2f - Update init.pp [20:06:20] [02puppet] 07paladox synchronize pull request 03#1732: varnish: Move stunnel configuration to its own class - 13https://git.io/JYQhL [20:06:29] [02puppet] 07paladox closed pull request 03#1732: varnish: Move stunnel configuration to its own class - 13https://git.io/JYQhL [20:06:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JYQhd [20:06:32] [02miraheze/puppet] 07paladox 0311ca17e - varnish: Move stunnel configuration to its own class (#1732) [20:06:33] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [20:06:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [20:07:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYQhj [20:07:24] [02miraheze/puppet] 07paladox 03f885984 - varnish::stunnel4: Removed miss params [20:12:04] [02puppet] 07paladox closed pull request 03#1718: puppetserver: +ssh key - 13https://git.io/JmVRO [20:17:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch-6 [+21/-13/±93] 13https://git.io/JY7e8 [20:17:24] [02miraheze/puppet] 07paladox 03b5a5e40 - Update apt module to 8.0 [20:17:26] [02puppet] 07paladox created branch 03patch-6 - 13https://git.io/vbiAS [20:17:36] [02puppet] 07paladox opened pull request 03#1733: Update apt module to 8.0 - 13https://git.io/JY7e4 [20:22:10] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:22:37] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:22:50] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:23:15] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:24:14] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:24:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch-6 [+20/-12/±94] 13https://git.io/JY7vs [20:24:31] [02miraheze/puppet] 07paladox 03fbf629d - Update apt module to 8.0 [20:24:33] [02puppet] 07paladox synchronize pull request 03#1733: Update apt module to 8.0 - 13https://git.io/JY7e4 [20:24:34] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:25:04] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:27:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:27:55] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:27:58] PROBLEM - gluster4 Puppet on gluster4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:31:41] PROBLEM - dbbackup1 Puppet on dbbackup1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:32:07] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:32:50] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:36:10] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:40:29] [02puppet] 07paladox closed pull request 03#1733: Update apt module to 8.0 - 13https://git.io/JY7e4 [20:40:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+20/-12/±94] 13https://git.io/JY7J4 [20:40:32] [02miraheze/puppet] 07paladox 0325b1c88 - Update apt module to 8.0 (#1733) [20:40:34] [02miraheze/puppet] 07paladox deleted branch 03patch-6 [20:40:35] [02puppet] 07paladox deleted branch 03patch-6 - 13https://git.io/vbiAS [20:48:19] 33 jobs outstanding on testwiki and 20 jobs on metawiki, the former of which have been outstanding for the past 30 minutes. [20:48:37] * dmehus looks forward to completion of that global jobs view task in Grafana :D [20:50:37] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:50:51] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:51:14] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:52:15] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:52:32] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:53:04] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:53:58] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [20:55:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:55] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:58:07] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:59:40] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:00:51] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:17:11] dmehus: still issue [21:17:16] ? [21:17:48] RhinosF1, let me check [21:18:39] RhinosF1, yes, jobs outstanding has grown on both wikis, 40 now on TestWiki [21:18:53] Fun [21:19:00] Part of the jobs I suspect could be Universal_Omega's global user page updates :D [21:19:08] that always backlogs things [21:19:25] Oh great [21:20:21] I'll get around to https://phabricator.miraheze.org/T6974 tomorrow anyway [21:20:22] [ ⚓ T6974 Jobs Statistics in Grafana ] - phabricator.miraheze.org [21:20:32] JohnLewis, cool, thanks :) [21:21:30] Ty JohnLewis [21:21:39] dmehus: I'll run the script in a sec [21:21:40] * dmehus wonders if there's a function in AbuseFilter whereby we could limit the number of user page edits to 1 every 10 minutes per unique user page [21:21:46] RhinosF1, ack, ty [21:22:16] you might need to set a max memory allocation, though [21:22:19] It just doesn't garbage collect well [21:22:47] In an ideal world, every edit wouldn't trigger a job. Only the latest. [21:22:55] yeah [21:22:57] Well trigger a job that runs [21:23:16] heh one idea is we could enable FlaggedRevs on Loginwiki [21:23:24] No [21:23:35] then only the currently approved revision would trigger the user page update [21:23:43] I don't even want to think what mess that would cause [21:24:14] It's the most complex, under tested, pile of technical debt anyone could invent [21:24:25] why? what mess? We could configure it to apply only to `User:` namespace [21:24:55] It's code base is hell [21:25:05] RhinosF1, well sort of. English Wikipedia does it right by having it apply only to pages protected as `pendingchangesprotected` [21:25:06] There's a reason half its features will go with 1.36 [21:25:30] well I mean Wikimedia already is on 1.36 so... [21:25:51] dmehus: if it weren't wikimedia deployed, it wouldn't have passed a security review. They're merging patches over the next few weeks. [21:26:12] Any reviewer would be insane to look at that and survive [21:26:51] maybe, hard to say it wouldn't have passed a security review, though [21:27:21] There's no way it would pass a wikimedia deployment for a new extension [21:27:44] And I doubt miraheze because it'd be too messy [21:27:53] It wouldn't even get its code uploaded to gerrit [21:28:04] hrm? how's that though? It is Wikimedia deployed [21:29:05] I agree it's messy, though. I personally think we should've set it up by default with a special protection level [21:29:09] Because it's already deployed [21:29:13] Standards change [21:29:20] okay [21:29:57] It had no proper maintainer beyond when it blocked the train until Amir took over a few weeks ago and started deleting half its code [21:30:25] oh [21:30:41] wow, that's quite surprising for an extension used on English Wikipedia [21:31:14] Only because of history [21:31:34] I wonder if English Wikipedia might be forced to do away with pending changes protection by virtue of the extension being demised then, without their community having any say in the matter [21:32:56] No they're keeping the basic features [21:33:07] English Wikipedia shouldn't change much [21:33:17] There's wikis that use more complex setups for it [21:33:24] And different interfaces [21:34:03] dmehus: meta jobs 1 [21:34:25] Login 0 [21:34:26] RhinosF1, weird. I wonder why TestWiki's jobs are deprioritized [21:34:48] Because I'm running meta,login,test [21:34:54] oh [21:34:57] Then all wikis [21:35:04] I didn't know you were running jobs right now [21:35:10] sorry thought you were just checking jobs [21:35:17] !log running jobs [21:35:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:26] Testwiki 0 [21:35:32] ah, thanks [21:36:34] All wikis running now [21:38:01] ah, thanks [21:43:31] * dmehus just got caught up on Canadian business news from the past several weeks and is happy to hear that Shopify has lost its standing as the largest Canadian company by stock market value again [21:46:03] paladox: https://graylog.miraheze.org/search?q=mediawiki_channel%3AMatomoAnalytics&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=300 [21:46:10] wikis missing from matomo again [21:46:23] you can add them using the script in MA maintenance folder [21:47:13] paladox: can we look at a script that can compare matomo to the cw_wikis table? [21:47:24] I'm fighting job queues am [21:47:26] am [21:47:27] atm [21:50:07] I mean you can [21:50:18] if you want [21:50:19] RhinosF1, I'm guessing this is the maintenance script (https://github.com/miraheze/MatomoAnalytics/blob/master/maintenance/modifyMatomo.php)? [21:50:19] [ MatomoAnalytics/modifyMatomo.php at master · miraheze/MatomoAnalytics · GitHub ] - github.com [21:50:27] that script only adds ids [21:50:30] or removes them [21:51:51] dmehus: yes [21:53:07] dmehus: I have runJobs waiting for meta&login new jobs now in a loop to keep it under control [21:53:15] if they go above zero then panic [21:57:41] RhinosF1 https://phabricator.miraheze.org/P401 should be what you want [21:57:42] [ ✎ P401 (An Untitled Masterwork) ] - phabricator.miraheze.org [22:03:20] paladox: [d0dd9d3a3051fae22be7150a] [no req] MWException from line 129 of /srv/mediawiki/w/includes/MagicWord.php: Error: invalid magic word 'getshortdesc' [22:03:23] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JY7m4 [22:03:25] [02miraheze/MatomoAnalytics] 07paladox 0392380d6 - Change Matomo to use makeGlobalKey rather than makeKey [22:03:26] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fN4LT [22:03:28] [02MatomoAnalytics] 07paladox opened pull request 03#45: Change Matomo to use makeGlobalKey rather than makeKey - 13https://git.io/JY7mB [22:03:32] [02MatomoAnalytics] 07paladox closed pull request 03#43: Change Matomo to use makeGlobalKey rather than makeKey - 13https://git.io/Jq2Np [22:03:33] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 [22:03:35] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fN4LT [22:03:48] RhinosF1 where is that? [22:04:18] paladox: running your script after sticking it in my homedir [22:04:28] miraheze/MatomoAnalytics - paladox the build passed. [22:04:56] is this on test3? [22:05:07] paladox: test3wiki but from jbr3 [22:05:38] 4 even [22:06:19] maybe lc needs running? [22:06:55] !log rebuild lc on jobrunner[34] [22:06:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:07:30] paladox: that's been done like 5 times for that issue. [22:07:51] that extension is trouble [22:08:57] i guess its because https://github.com/StarCitizenTools/mediawiki-extensions-ShortDescription/blob/master/extension.json is missing a magic file... [22:08:58] [ mediawiki-extensions-ShortDescription/extension.json at master · StarCitizenTools/mediawiki-extensions-ShortDescription · GitHub ] - github.com [22:09:26] can we get a fix [22:09:28] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:10:34] https://github.com/StarCitizenTools/mediawiki-extensions-ShortDescription/pull/5 [22:10:34] [ Add missing ExtensionMessagesFiles by paladox · Pull Request #5 · StarCitizenTools/mediawiki-extensions-ShortDescription · GitHub ] - github.com [22:11:47] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [22:12:03] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is UNKNOWN: NRPE: Unable to read output [22:18:14] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JY7YV [22:18:16] [02miraheze/mediawiki] 07paladox 03f14d927 - Update ShortDescription [22:19:20] paladox: see their comment, they reverted [22:21:06] well i guess no one ran i18n then... [22:21:12] !log rebuild lc on mw* and jobrunner* [22:21:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:33] paladox: ping me when it's ran [22:23:30] [02MatomoAnalytics] 07paladox closed pull request 03#45: Change Matomo to use makeGlobalKey rather than makeKey - 13https://git.io/JY7mB [22:23:32] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-2 [22:23:33] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fN4LT [22:24:34] https://www.irccloud.com/pastebin/gt7INgL5/ [22:24:34] [ Snippet | IRCCloud ] - www.irccloud.com [22:24:37] paladox: ^ [22:24:51] apart from needing new lines [22:24:54] lgtm [22:24:58] so it works now [22:25:01] ye [22:25:08] you can easily add a new line \n [22:25:16] i can [22:26:06] RhinosF1, ack, okay sounds good [22:26:52] [02miraheze/MatomoAnalytics] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+1/-0/±0] 13https://git.io/JY7O0 [22:26:54] [02miraheze/MatomoAnalytics] 07RhinosF1 03b89aaa0 - Create addMissingMatomos.php [22:26:55] [02MatomoAnalytics] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fN4LT [22:27:13] [02MatomoAnalytics] 07RhinosF1 opened pull request 03#46: Create addMissingMatomos.php - 13https://git.io/JY7Ou [22:27:15] [02MatomoAnalytics] 07RhinosF1 assigned pull request 03#46: Create addMissingMatomos.php - 13https://git.io/JY7Ou [22:27:19] paladox: ^ [22:28:15] miraheze/MatomoAnalytics - RhinosF1 the build passed. [22:28:37] seems fine [22:28:40] feel free to merge [22:28:47] don't forget a version bump [22:28:48] and changelog [22:28:52] [02MatomoAnalytics] 07RhinosF1 closed pull request 03#46: Create addMissingMatomos.php - 13https://git.io/JY7Ou [22:28:53] [02miraheze/MatomoAnalytics] 07RhinosF1 pushed 032 commits to 03master [+2/-0/±0] 13https://git.io/JY7O6 [22:28:55] [02miraheze/MatomoAnalytics] 07RhinosF1 031d7d743 - Merge pull request #46 from miraheze/RhinosF1-patch-1 [22:28:56] [02miraheze/MatomoAnalytics] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [22:28:58] [02MatomoAnalytics] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/fN4LT [22:29:00] oh [22:29:18] What happens if CreateWiki isn’t installed and that is ran? [22:29:53] miraheze/MatomoAnalytics - RhinosF1 the build passed. [22:30:07] it fails JohnLewis [22:30:18] Non cleanly [22:30:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 6.03, 4.36 [22:30:32] it should fail cleanly with useful error reporting [22:31:11] [02miraheze/MatomoAnalytics] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JY7OF [22:31:13] [02miraheze/MatomoAnalytics] 07RhinosF1 03918e187 - Update CHANGELOG [22:31:14] [02MatomoAnalytics] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fN4LT [22:31:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.71, 7.38, 5.10 [22:31:21] [02MatomoAnalytics] 07RhinosF1 opened pull request 03#47: Update CHANGELOG & Version - 13https://git.io/JY7ON [22:31:37] [02miraheze/MatomoAnalytics] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JY7Ox [22:31:39] [02miraheze/MatomoAnalytics] 07RhinosF1 03080dbf9 - Update extension.json [22:31:40] [02MatomoAnalytics] 07RhinosF1 synchronize pull request 03#47: Update CHANGELOG & Version - 13https://git.io/JY7ON [22:31:41] JohnLewis, by useful error reporting, do you mean helpful error reports rather than fatal DB exception errors?, or are you speaking mainly to the error logging? [22:31:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.08, 6.61, 4.89 [22:32:23] miraheze/MatomoAnalytics - RhinosF1 the build passed. [22:32:32] JohnLewis: i assume $config->get( 'CreateWikiDatabase' ) would return None/False? [22:32:46] miraheze/MatomoAnalytics - RhinosF1 the build passed. [22:33:15] Yes, but that wouldn’t be needed as maintenance has an extension requirement check in it [22:33:52] JohnLewis: how? [22:34:13] https://www.websiteplanet.com/blog/miraheze-interview/ \o/ [22:34:13] [ Build Your Own Wiki Platform And Grow Your Community With Miraheze ] - www.websiteplanet.com [22:34:17] Go read the docs for the maintenance class as I don’t know what it is off by heart [22:34:31] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [22:35:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.75, 5.79 [22:36:13] JohnLewis: https://doc.wikimedia.org/mediawiki-core/master/php/classMaintenance.html#aebd4192b237706bbb9c524188934bd93 [22:36:14] [ MediaWiki: Maintenance Class Reference ] - doc.wikimedia.org [22:36:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.89, 6.27, 5.09 [22:37:21] Mhm? [22:37:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.84, 6.24, 5.34 [22:39:19] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.65, 6.68, 5.86 [22:43:01] JohnLewis: the jobrunners are still massively behind too [22:43:30] What kind of jobs and what numbers? [22:43:54] JohnLewis: mostly GUP ones and my runJobs is only on b wikis [22:44:29] So you’re running jobs without having collect data before hand? [22:45:26] i have the count in the personal bar, testwiki has 42 jobs waiting to run. [22:46:10] i could see login+meta climbing constantly until i set my shell to run it with --wait because they more important [22:46:11] I’ve been asking for 2 months now for a breakdown of the number of jobs per type per wiki when they’re high [22:46:31] I’m genuinely unsure how I can be more clear [22:55:57] dmehus: I'm going to sleep but aggressively ping if jobs get out of control. I have screens running. [22:59:54] RhinosF1, okay [23:03:07] JohnLewis: how much time before you're afk (sleep, other work, etc)? I need an SRE for a test regarding https://phabricator.miraheze.org/T5877, but not wise to do that without an SRE being around [23:03:09] [ ⚓ T5877 Revise MariaDB backup strategy ] - phabricator.miraheze.org [23:42:07] night [23:46:32] night, SPF|Cloud