[00:03:55] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:04:37] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:04:50] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:38] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:42] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:17:13] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYF9V [00:17:14] [02miraheze/puppet] 07JohnFLewis 03a4a8da3 - remove claimed and delayed data collection [00:25:19] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYFHc [00:25:20] [02miraheze/puppet] 07JohnFLewis 03adcfd9f - wiki:jobqueue not global:jobqueue [00:56:04] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.08, 1.98, 1.58 [00:58:04] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.12, 1.62, 1.49 [01:19:30] https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?orgId=1 [01:19:31] [ Grafana ] - grafana.miraheze.org [01:20:44] (fun fact we have ~51.4k unclaimed, lost jobs that made the Redis install extremely large and sluggish at times) [01:20:51] well, *had :P [01:23:33] ^ cc dmehus Reception123 paladox Universal_Omega [01:25:41] page loading feels a lot faster now, assuming that is why [01:25:45] nice 👍 [01:25:51] I was going to introduce logging of 'claimed' status as code originally shows, but abandoned it because we'd only have a maximum of 4 claimed jobs ever - as we only run 4 general job queue processes so I felt it added no value and added an extra few commands [01:28:15] @dorito I'll have to wait 24 hours to see any significant changes in recorded performance but I heavily doubt it would correlate, but I can see why it may have a very very minor impact - just not enough to say it would be 'noticeable' but I'm open to be disproven by the data [01:28:54] I might just be imagining things, idk just feels faster compared to yesterday and the day before [01:29:09] Cause I was getting >5 second loads sometimes and now its consistently <2 [01:33:06] JohnLewis, oh wow, what was the composition of those unclaimed jobs and how far back did they date? [01:33:43] Oh, GlobalUserPage jobs [01:34:04] yeah... that doesn't surprise me at all. I figured that'd be the most of them [01:36:03] * dmehus just approved a wiki...checking jobqueue :P [01:37:35] There was a time when we just ‘dropped’ all global user page cache jobs. Obviously whoever did that, just broke the link and didn’t actually delete the jobs :) [01:44:59] JohnLewis, ah that makes sense. So those 51,400 GUP cache jobs weren't all Universal_Omega's GUP updates then heh [01:45:39] yeah... this will be a lot more helpful now having this data so readily accessible :) [01:47:36] dmehus: yeah, don't think I created that many jobs. Though I did do alot, and will never again... whoops... [01:49:42] 2nd graph in particular as it allows us easily to see proportions of jobs. If most of the jobs aren’t accounted for, then it’ll signal a new intensive job we need to be weary of [01:49:54] Universal_Omega, yeah, heh. It doesn't seem like we lost anything by dropping the global user page jobs, I suspect, eh JohnLewis? [01:50:16] > 2nd graph in particular as it allows us easily to see proportions of jobs. If most of the jobs aren’t accounted for, then it’ll signal a new intensive job we need to be weary of [01:50:16] yeah, exactly [01:51:04] Well they were 51k jobs that just take up memory. They were never going to run [01:53:52] yeah... would there be end user impacts, or would we likely have long resolved these by now (i.e., by manually purging local copies of user pages)? [01:55:43] End user impact definitely because the job to update the cache was never ran [01:58:16] ah [02:20:47] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.94, 1.54, 1.23 [02:22:44] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.97, 1.32, 1.18 [03:25:45] [02ManageWiki] 07Universal-Omega opened pull request 03#264: Don't log if errors exist (Will hopefully fix T7069) - 13https://git.io/JYbmf [03:26:41] miraheze/ManageWiki - Universal-Omega the build passed. [03:37:25] [02ManageWiki] 07Universal-Omega synchronize pull request 03#264: Don't log if errors exist (Will hopefully fix T7069) - 13https://git.io/JYbmf [03:37:59] [02ManageWiki] 07Universal-Omega edited pull request 03#264: ManageWikiPermissions: Don't log if errors exist (to fix T7069) - 13https://git.io/JYbmf [03:38:28] miraheze/ManageWiki - Universal-Omega the build passed. [04:40:22] [02ManageWiki] 07Reception123 closed pull request 03#264: ManageWikiPermissions: Don't log if errors exist (to fix T7069) - 13https://git.io/JYbmf [04:40:23] [02miraheze/ManageWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYbCW [04:40:25] [02miraheze/ManageWiki] 07Universal-Omega 035f50e32 - ManageWikiPermissions: Don't log if errors exist (to fix T7069) (#264) [04:41:23] miraheze/ManageWiki - Reception123 the build passed. [04:42:00] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbCa [04:42:02] [02miraheze/mediawiki] 07Reception123 03e9da12f - Update ManageWiki [04:52:00] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JYbWH [04:52:02] [02miraheze/ssl] 07Reception123 031171846 - add wiki.umbranet.mandelbrotgames.com cert [04:52:03] [02miraheze/ssl] 07Reception123 03fd4a66d - add matttest.mtwiki.cf cert [05:50:41] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbuq [05:50:43] [02miraheze/mediawiki] 07Reception123 03686b5f6 - Update DPLForum [05:51:05] ^ @dorito [05:51:12] 👍 [05:57:41] [02mediawiki] 07Reception123 closed pull request 03#1402: Bump y18n from 3.2.1 to 3.2.2 - 13https://git.io/JYVQ6 [05:57:42] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbuh [05:57:44] [02miraheze/mediawiki] 07dependabot[bot] 0368a7775 - Bump y18n from 3.2.1 to 3.2.2 (#1402) [05:57:45] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/npm_and_yarn/y18n-3.2.2 [05:57:47] [02mediawiki] 07Reception123 deleted branch 03dependabot/npm_and_yarn/y18n-3.2.2 - 13https://git.io/vbL5b [06:09:18] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03revert-1402-dependabot/npm_and_yarn/y18n-3.2.2 [+0/-0/±1] 13https://git.io/JYbgc [06:09:20] [02miraheze/mediawiki] 07Reception123 0313bb3d1 - Revert "Bump y18n from 3.2.1 to 3.2.2 (#1402)" [06:09:21] [02mediawiki] 07Reception123 created branch 03revert-1402-dependabot/npm_and_yarn/y18n-3.2.2 - 13https://git.io/vbL5b [06:09:23] [02mediawiki] 07Reception123 opened pull request 03#1443: Revert "Bump y18n from 3.2.1 to 3.2.2" - 13https://git.io/JYbgl [06:09:27] [02mediawiki] 07Reception123 closed pull request 03#1443: Revert "Bump y18n from 3.2.1 to 3.2.2" - 13https://git.io/JYbgl [06:09:28] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbg8 [06:09:30] [02miraheze/mediawiki] 07Reception123 032ce9682 - Revert "Bump y18n from 3.2.1 to 3.2.2 (#1402)" (#1443) [06:09:31] [02miraheze/mediawiki] 07Reception123 deleted branch 03revert-1402-dependabot/npm_and_yarn/y18n-3.2.2 [06:09:33] [02mediawiki] 07Reception123 deleted branch 03revert-1402-dependabot/npm_and_yarn/y18n-3.2.2 - 13https://git.io/vbL5b [06:27:59] RECOVERY - services4 APT on services4 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [06:53:03] RECOVERY - services3 APT on services3 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [07:17:19] [02mediawiki] 07RhinosF1 labeled pull request 03#1436: Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` - 13https://git.io/JYSNO [07:18:06] [02mediawiki] 07RhinosF1 labeled pull request 03#1434: Bump extensions/CommentStreams from `bbae251` to `ed3fc73` - 13https://git.io/JYSNt [07:19:09] [02mediawiki] 07RhinosF1 labeled pull request 03#1433: Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` - 13https://git.io/JYSNk [07:19:41] [02mediawiki] 07RhinosF1 labeled pull request 03#1430: Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` - 13https://git.io/JYSbj [07:20:06] [02mediawiki] 07RhinosF1 labeled pull request 03#1427: Bump extensions/TemplateWizard from `9b8d627` to `439da85` - 13https://git.io/JYSbd [07:20:32] [02mediawiki] 07RhinosF1 labeled pull request 03#1425: Bump skins/MinervaNeue from `97729f6` to `4f16566` - 13https://git.io/JYSby [07:20:52] [02mediawiki] 07RhinosF1 labeled pull request 03#1422: Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` - 13https://git.io/JYSb6 [07:21:12] [02mediawiki] 07RhinosF1 labeled pull request 03#1421: Bump skins/Cosmos from `e3b643f` to `e54190f` - 13https://git.io/JYSbr [07:21:45] [02mediawiki] 07RhinosF1 labeled pull request 03#1417: Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` - 13https://git.io/JYSbB [07:21:57] [02mediawiki] 07RhinosF1 labeled pull request 03#1416: Bump extensions/Math from `a412f37` to `f849125` - 13https://git.io/JYSb8 [07:22:59] [02mediawiki] 07RhinosF1 labeled pull request 03#1414: Bump extensions/RelatedArticles from `8a38280` to `954f1b6` - 13https://git.io/JYSbn [07:23:15] [02mediawiki] 07RhinosF1 labeled pull request 03#1413: Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` - 13https://git.io/JYSbG [07:23:32] [02mediawiki] 07RhinosF1 labeled pull request 03#1411: Bump extensions/Cite from `52f3cf3` to `bc16b05` - 13https://git.io/JYSbq [07:23:57] [02mediawiki] 07RhinosF1 labeled pull request 03#1408: Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` - 13https://git.io/JYSbf [07:24:46] [02mediawiki] 07RhinosF1 labeled pull request 03#1404: Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` - 13https://git.io/JYSFd [07:27:41] [02mediawiki] 07RhinosF1 closed pull request 03#1430: Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` - 13https://git.io/JYSbj [07:27:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbPQ [07:27:44] [02miraheze/mediawiki] 07dependabot[bot] 032933160 - Bump extensions/Newsletter from `7e9cd56` to `1b6a18c` (#1430) [07:27:46] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/Newsletter-1b6a18c - 13https://git.io/vbL5b [07:27:47] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/Newsletter-1b6a18c [07:28:01] [02mediawiki] 07RhinosF1 closed pull request 03#1427: Bump extensions/TemplateWizard from `9b8d627` to `439da85` - 13https://git.io/JYSbd [07:28:03] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbPF [07:28:04] [02miraheze/mediawiki] 07dependabot[bot] 0398e331f - Bump extensions/TemplateWizard from `9b8d627` to `439da85` (#1427) [07:28:06] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/TemplateWizard-439da85 - 13https://git.io/vbL5b [07:28:07] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/TemplateWizard-439da85 [07:28:20] [02mediawiki] 07RhinosF1 closed pull request 03#1425: Bump skins/MinervaNeue from `97729f6` to `4f16566` - 13https://git.io/JYSby [07:28:22] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbPA [07:28:23] [02miraheze/mediawiki] 07dependabot[bot] 03d7326fa - Bump skins/MinervaNeue from `97729f6` to `4f16566` (#1425) [07:28:25] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/skins/MinervaNeue-4f16566 [07:28:26] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/skins/MinervaNeue-4f16566 - 13https://git.io/vbL5b [07:28:32] [02mediawiki] 07RhinosF1 closed pull request 03#1422: Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` - 13https://git.io/JYSb6 [07:28:34] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbPh [07:28:35] [02miraheze/mediawiki] 07dependabot[bot] 038aed2ef - Bump extensions/ProofreadPage from `f97be2c` to `63a93d5` (#1422) [07:28:37] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/ProofreadPage-63a93d5 - 13https://git.io/vbL5b [07:28:38] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/ProofreadPage-63a93d5 [07:28:48] [02mediawiki] 07RhinosF1 closed pull request 03#1417: Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` - 13https://git.io/JYSbB [07:28:50] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbXe [07:28:51] [02miraheze/mediawiki] 07dependabot[bot] 03ac0416b - Bump extensions/AbuseFilter from `f528f3d` to `6eb8dd9` (#1417) [07:28:53] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/AbuseFilter-6eb8dd9 - 13https://git.io/vbL5b [07:28:54] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/AbuseFilter-6eb8dd9 [07:29:06] [02mediawiki] 07RhinosF1 closed pull request 03#1416: Bump extensions/Math from `a412f37` to `f849125` - 13https://git.io/JYSb8 [07:29:08] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbXT [07:29:09] [02miraheze/mediawiki] 07dependabot[bot] 0318184e3 - Bump extensions/Math from `a412f37` to `f849125` (#1416) [07:29:11] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/Math-f849125 [07:29:12] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/Math-f849125 - 13https://git.io/vbL5b [07:29:16] [02mediawiki] 07RhinosF1 closed pull request 03#1414: Bump extensions/RelatedArticles from `8a38280` to `954f1b6` - 13https://git.io/JYSbn [07:29:18] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbXL [07:29:19] [02miraheze/mediawiki] 07dependabot[bot] 03c67cc6a - Bump extensions/RelatedArticles from `8a38280` to `954f1b6` (#1414) [07:29:21] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/RelatedArticles-954f1b6 [07:29:22] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/RelatedArticles-954f1b6 - 13https://git.io/vbL5b [07:29:27] [02mediawiki] 07RhinosF1 closed pull request 03#1413: Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` - 13https://git.io/JYSbG [07:29:29] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbXq [07:29:30] [02miraheze/mediawiki] 07dependabot[bot] 03dea9071 - Bump extensions/TwoColConflict from `0f00dc0` to `29639ad` (#1413) [07:29:34] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/TwoColConflict-29639ad - 13https://git.io/vbL5b [07:29:35] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/TwoColConflict-29639ad [07:29:38] [02mediawiki] 07RhinosF1 closed pull request 03#1411: Bump extensions/Cite from `52f3cf3` to `bc16b05` - 13https://git.io/JYSbq [07:29:40] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbXY [07:29:41] [02miraheze/mediawiki] 07dependabot[bot] 038f02b88 - Bump extensions/Cite from `52f3cf3` to `bc16b05` (#1411) [07:29:43] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/Cite-bc16b05 [07:29:44] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/Cite-bc16b05 - 13https://git.io/vbL5b [07:29:48] [02mediawiki] 07RhinosF1 closed pull request 03#1408: Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` - 13https://git.io/JYSbf [07:29:49] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbX3 [07:29:51] [02miraheze/mediawiki] 07dependabot[bot] 031deda15 - Bump extensions/RevisionSlider from `8bdb906` to `d1a6af2` (#1408) [07:29:52] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/RevisionSlider-d1a6af2 [07:29:54] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/RevisionSlider-d1a6af2 - 13https://git.io/vbL5b [07:29:56] [02mediawiki] 07RhinosF1 closed pull request 03#1404: Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` - 13https://git.io/JYSFd [07:29:58] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbXZ [07:29:59] [02miraheze/mediawiki] 07dependabot[bot] 038cfccd8 - Bump extensions/ElectronPdfService from `89a3856` to `0d21aed` (#1404) [07:30:01] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/ElectronPdfService-0d21aed - 13https://git.io/vbL5b [07:30:02] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/ElectronPdfService-0d21aed [07:37:25] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:38:18] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:38:38] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:38:49] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:38:54] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:39:22] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:39:28] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [07:40:46] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JYbML [07:40:48] [02miraheze/mediawiki] 07RhinosF1 036167fed - [DNM]: Dependabot: Switch to REL1_36 [07:40:49] [02mediawiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbL5b [07:40:58] [02mediawiki] 07RhinosF1 opened pull request 03#1444: [DNM]: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMt [07:41:04] [02mediawiki] 07RhinosF1 assigned pull request 03#1444: [DNM]: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMt [07:45:15] [02mediawiki] 07Universal-Omega commented on pull request 03#1444: [DNM]: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMD [07:46:03] [02mediawiki] 07RhinosF1 commented on pull request 03#1444: [DNM]: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMF [07:46:44] Reception123: can you run "cd /srv/mediawiki/w/extensions && sudo -u www-data git reset --hard origin/REL1_35 && sudo -u www-data git pull ; sudo -u www-data git submodule update" on mw* [07:47:02] I just did test3/jbr* [07:47:24] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:49:45] [02mediawiki] 07RhinosF1 edited pull request 03#1444: [DNM]: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMt [08:03:22] will do [08:04:16] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [08:04:54] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:05:21] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:05:26] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:06:15] ok [08:06:25] [02mediawiki] 07dependabot[bot] opened pull request 03#1445: Bump y18n from 3.2.1 to 3.2.2 - 13https://git.io/JYbSy [08:06:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/y18n-3.2.2 [+0/-0/±1] 13https://git.io/JYbSS [08:06:28] [02miraheze/mediawiki] 07dependabot[bot] 03afca2b9 - Bump y18n from 3.2.1 to 3.2.2 [08:06:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/y18n-3.2.2 - 13https://git.io/vbL5b [08:06:31] [02mediawiki] 07dependabot[bot] labeled pull request 03#1445: Bump y18n from 3.2.1 to 3.2.2 - 13https://git.io/JYbSy [08:06:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1445: Bump y18n from 3.2.1 to 3.2.2 - 13https://git.io/JYbSy [08:06:40] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [08:06:48] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:06:49] RhinosF1: don't forget to log though :) [08:06:51] !log cd /srv/mediawiki/w/extensions && sudo -u www-data git reset --hard origin/REL1_35 && sudo -u www-data git pull ; sudo -u www-data git submodule update on mw* [08:06:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:07:23] you did it [08:08:10] [02mediawiki] 07RhinosF1 closed pull request 03#1436: Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` - 13https://git.io/JYSNO [08:08:11] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbSA [08:08:13] [02miraheze/mediawiki] 07dependabot[bot] 034fc633f - Bump extensions/ApprovedRevs from `f66b77f` to `5668e4e` (#1436) [08:08:14] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-5668e4e [08:08:16] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-5668e4e - 13https://git.io/vbL5b [08:08:19] [02mediawiki] 07RhinosF1 closed pull request 03#1434: Bump extensions/CommentStreams from `bbae251` to `ed3fc73` - 13https://git.io/JYSNt [08:08:20] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYbSj [08:08:22] [02miraheze/mediawiki] 07dependabot[bot] 03d4edc01 - Bump extensions/CommentStreams from `bbae251` to `ed3fc73` (#1434) [08:08:23] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-ed3fc73 [08:08:25] yeah, mean in general [08:08:25] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-ed3fc73 - 13https://git.io/vbL5b [08:08:45] [02mediawiki] 07RhinosF1 closed pull request 03#1433: Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` - 13https://git.io/JYSNk [08:08:47] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYb9J [08:08:48] [02miraheze/mediawiki] 07dependabot[bot] 033e4bf64 - Bump extensions/CreateWiki from `0c9d862` to `e60c4a5` (#1433) [08:08:50] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/CreateWiki-e60c4a5 [08:08:51] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_35/extensions/CreateWiki-e60c4a5 - 13https://git.io/vbL5b [08:08:56] [02mediawiki] 07RhinosF1 closed pull request 03#1421: Bump skins/Cosmos from `e3b643f` to `e54190f` - 13https://git.io/JYSbr [08:08:57] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYb9I [08:08:59] [02miraheze/mediawiki] 07dependabot[bot] 034af1b67 - Bump skins/Cosmos from `e3b643f` to `e54190f` (#1421) [08:09:10] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Cosmos-e54190f [08:09:11] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Cosmos-e54190f - 13https://git.io/vbL5b [08:10:33] !log mergemessagelistfiles and rebuild i18n on test3 then jbr* then mw* (START) [08:10:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:13] [02mediawiki] 07RhinosF1 closed pull request 03#1445: Bump y18n from 3.2.1 to 3.2.2 - 13https://git.io/JYbSy [08:11:15] [02mediawiki] 07dependabot[bot] commented on pull request 03#1445: Bump y18n from 3.2.1 to 3.2.2 - 13https://git.io/JYb9B [08:11:16] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/npm_and_yarn/y18n-3.2.2 [08:11:18] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/npm_and_yarn/y18n-3.2.2 - 13https://git.io/vbL5b [08:14:18] [02mediawiki] 07Reception123 commented on pull request 03#1444: [DNM]: Dependabot: Switch to REL1_36 - 13https://git.io/JYb9x [08:43:12] !log mergemessagelistfiles and rebuild i18n on test3 then jbr* then mw* (END) [08:43:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:40:06] PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: connect to address stablestate.org and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [09:46:46] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Mon 05 Jul 2021 13:02:00 GMT +0000. [09:55:38] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.31, 4.26, 2.19 [09:56:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.236.250/cpweb, 2001:41d0:800:1bbd::4/cpweb [09:56:38] f [09:57:37] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.75, 4.73, 2.64 [09:58:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:19:20] JohnLewis: around for a question regarding the new jobs metrics? [14:25:10] Reception123: missed you message before but I am now [14:33:32] JohnLewis: how are there 3 separate instances of GUP cache updates running (https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue) when there's only been one edit on loginwiki today? [14:33:33] [ Grafana ] - grafana.miraheze.org [14:33:51] and it's also weird that in between the increases it shows a static number of jobs [14:39:55] Jobs have 4 possible states, we only monitor 2 of them [14:40:43] but where is the sudden increase coming from then if there's only been one edit (the first increase)? [14:40:46] So if a job fails on its initial run, it'll go into a delayed queue, then be requeued later [14:41:01] When it is requeued, it's unclaimed [14:45:58] oh, that's interesting. That could be interesting if we see a lot of jobs being moved into the delayed queue [14:46:11] oh, I didn't know that [14:48:04] https://www.irccloud.com/pastebin/DawL4e39/ [14:48:05] [ Snippet | IRCCloud ] - www.irccloud.com [14:48:17] Might also account for the static number of jobs [15:21:58] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JYNpI [15:22:00] [02miraheze/mw-config] 07RhinosF1 03f0fc1e2 - Flaggedrevs: don't use 1.36 removed config for new installs [15:22:01] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [15:22:08] [02mw-config] 07RhinosF1 opened pull request 03#3817: Flaggedrevs: don't use 1.36 removed config for new installs - 13https://git.io/JYNpt [15:23:14] miraheze/mw-config - RhinosF1 the build passed. [15:24:03] .mhca Vgutierrez [15:24:03] https://meta.miraheze.org/wiki/Special:CentralAuth/Vgutierrez [15:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYNp6 [15:25:14] [02miraheze/services] 07MirahezeSSLBot 036527b10 - BOT: Updating services config for wikis [15:40:37] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYAec [15:40:39] [02miraheze/puppet] 07JohnFLewis 032229561 - jobrunner: use Miraheze repo not Wikimedia [15:40:48] !log fork wikimedia/mediawiki-services-jobrunner repo [15:40:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:35] dmehus: renaming wikis isn't supported offically [16:41:10] RhinosF1, ah, yeah, true. That's what Reception123 means when he says they probably wouldn't like our upstream task since they don't support renaming a wiki's database name [16:41:24] Yes [16:41:42] There is a task for them restarting wiki renames [16:41:48] But it's not moved in a while [16:41:48] oh [16:44:05] It is something they want though [16:50:16] oh, that's good then; means it'll likely get done, eventually [16:54:27] Eventually [17:03:38] [02miraheze/jobrunner-service] 07JohnFLewis pushed 031 commit to 03master [+0/-2/±5] 13https://git.io/JYAYs [17:03:39] [02miraheze/jobrunner-service] 07JohnFLewis 03554519c - Clean up for PHP7.3 [17:12:09] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [17:34:24] Reception123: JobQueue peaked at 30k queued jobs about 14 minutes ago :P [17:36:14] JohnLewis: loginwiki [17:36:28] I know, I can see the data [17:36:34] Been about 6 edits [17:39:59] yikes [17:41:24] I'm not sure how possible it is via AF but I was thinking it may be useful to provide a warning to a user once they've edited their GUP more than 3 times a day [17:41:40] Surely we can at least recommend that they use preview rather than submit every time and create all these jobs [17:43:02] Reception123: https://www.mediawiki.org/wiki/Extension:ForcePreview / https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Force_preview + an edit notice [17:43:02] [ Extension:ForcePreview - MediaWiki ] - www.mediawiki.org [17:43:02] [ Manual:Force preview - MediaWiki ] - www.mediawiki.org [17:45:45] oh true, there's that option too [17:46:02] ^ dmehus as you were interested in this [17:51:14] Reception123, yeah we could do that, as I did consider that as well [17:54:12] RhinosF1, what ns ID is user namespace again? [17:54:19] Reception123: I’m taking the time now to give proper care and attention to our jobs process from a technical side, after I’ll start considering what changes we could make to our technical infrastructure with regards to MediaWiki capacity. Do you want me to create a task for these considerations so MW-SRE can consider them or would you prefer a paste proposal? [17:58:25] JohnLewis: I guess a task will be better so we can track it [18:02:29] Will do [18:14:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 6.36, 4.97 [18:16:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.16, 6.78, 5.27 [18:16:57] dmehus: https://www.mediawiki.org/wiki/Manual:Namespace#Built-in_namespaces [18:16:57] [ Manual:Namespace - MediaWiki ] - www.mediawiki.org [18:17:04] So 2 [18:18:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 6.65, 5.40 [18:20:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.88, 7.04, 5.21 [18:20:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.54, 8.20, 6.11 [18:22:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 7.29, 5.53 [18:28:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.57, 5.97, 5.61 [18:28:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.80, 7.00, 6.59 [18:32:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.54, 6.32, 6.43 [18:32:29] RhinosF1, yeah, thanks, I ended up finding not that page, but a different page on MediaWiki.org [18:32:36] Ok [18:32:45] yours was the better page [18:32:53] *should bookmark that* [18:32:55] Mw docs are a bit crazy [18:33:38] yeah [18:34:01] https://bash.toolforge.org/quip/AU7VVnI_6snAnmqnK_yu [18:34:02] [ quip - Quips ] - bash.toolforge.org [18:36:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 7.15, 6.73 [18:38:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.02, 6.74 [18:40:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.76, 6.28, 6.51 [18:46:01] JohnLewis: it would be nice if we could see the number of jobs processed per x and new jobs per x so we can see the difference [18:47:12] RhinosF1: anyone with edit access to Grafana can make that work (including you) [18:49:34] JohnLewis: where would I look [18:49:46] I don't even know if I've ever signed into grafana [18:50:50] Each panel should have an edit button on it for you, you should also be able to add new panels on Dashboards as well [18:51:09] https://grafana.miraheze.org/explore is a place you can test changes in a transient environment too [18:51:10] [ Grafana ] - grafana.miraheze.org [18:51:33] [02miraheze/jobrunner-service] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYA8y [18:51:34] [02miraheze/jobrunner-service] 07paladox 039cb972d - Avoid using deprecated phpredis::delete() alias [18:51:36] [02jobrunner-service] 07paladox created branch 03paladox-patch-1 - 13https://git.io/JYA8S [18:51:37] [02jobrunner-service] 07paladox opened pull request 03#1: Avoid using deprecated phpredis::delete() alias - 13https://git.io/JYA89 [18:51:45] [02jobrunner-service] 07paladox closed pull request 03#1: Avoid using deprecated phpredis::delete() alias - 13https://git.io/JYA89 [18:51:47] [02miraheze/jobrunner-service] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JYA87 [18:51:48] [02miraheze/jobrunner-service] 07paladox 034e54324 - Merge pull request #1 from miraheze/paladox-patch-1 [18:51:50] [02jobrunner-service] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/JYA8S [18:51:52] [02miraheze/jobrunner-service] 07paladox deleted branch 03paladox-patch-1 [18:52:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.52, 6.81, 5.74 [18:53:04] !log switch over jobrunner[34] to use our fork https://github.com/miraheze/jobrunner-service [18:53:04] [ GitHub - miraheze/jobrunner-service: JobRunner services on Miraheze. Originally a Wikimedia Foundation software available at https://github.com/wikimedia/mediawiki-services-jobrunner ] - github.com [18:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.21, 6.55, 5.78 [18:56:14] JohnLewis: where is jobrunner in the metrics list [18:56:35] redis_script_values [18:59:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 7.94, 6.87 [19:00:59] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.46, 6.80, 6.59 [19:03:35] Will fiddle later [19:13:23] hmm JobQueueError from line 778 of /srv/mediawiki/w/includes/jobqueue/JobQueueRedis.php: Redis server error: socket error on read socket [19:28:45] I have some troubles when implementing tempalte:main. I have already implemented module:main but the template:main still unfunctional. [19:28:57] https://cdn.discordapp.com/attachments/808001911868489748/829437610375512131/image0.png [19:37:39] Looks like the problem is with Template:Documentation instead of Template:Main. The latter is working as expected. [19:40:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYAEl [19:40:12] [02miraheze/services] 07MirahezeSSLBot 0372d0b6b - BOT: Updating services config for wikis [19:41:41] [02miraheze/jobrunner-service] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JYAE2 [19:41:42] [02miraheze/jobrunner-service] 07JohnFLewis 03aa692fe - Remove json_decode syntax error message (non-issue) [19:41:44] [02miraheze/jobrunner-service] 07JohnFLewis 035b3c13f - Merge remote-tracking branch 'origin/master' [19:49:46] So... what should I do to fix the problem ? [19:59:45] @Mister OwO You can fix it by copying over [[Template:Documentation]] and [[Module:Documentation]] to your wiki [19:59:46]

https://meta.miraheze.org/wiki/Template:Documentation https://meta.miraheze.org/wiki/Module:Documentation [19:59:47] [ Template:Documentation - Miraheze Meta ] - meta.miraheze.org [19:59:48] [ Module:Documentation - Miraheze Meta ] - meta.miraheze.org [20:02:50] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+2/-1/±3] 13https://git.io/JYAzR [20:02:51] [02miraheze/puppet] 07JohnFLewis 038bf8f9b - jobrunner: only run jobchron on one server [20:08:39] PROBLEM - jobrunner4 JobChron Service on jobrunner4 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:10:01] LocalGlobalUserPageCacheUpdateJobs seem to be the only one getting 'stuck', the others level out at 0 jobs. Let me looking further into that [20:10:29] !log run jobs on all wikis on jobrunner3, experimentation run to empty queue through a forced run [20:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:12] [[Module:Category handler]] [20:11:13]

https://meta.miraheze.org/wiki/Module:Category_handler [20:11:13] [ Module:Category handler - Miraheze Meta ] - meta.miraheze.org [20:12:41] [[Modlue:Yesno]] [20:12:42]

https://meta.miraheze.org/wiki/Modlue:Yesno?action=edit&redlink=1 [20:12:43] [ Modlue:Yesno - Miraheze Meta ] - meta.miraheze.org [20:13:49] [[Module:Yesno]] [20:13:50]

https://meta.miraheze.org/wiki/Module:Yesno [20:13:52] [ Module:Yesno - Miraheze Meta ] - meta.miraheze.org [20:16:10] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [20:16:51] [[Module:Documentation/config]] [20:16:52]

https://meta.miraheze.org/wiki/Module:Documentation/config [20:16:53] [ Module:Documentation/config - Miraheze Meta ] - meta.miraheze.org [20:18:55] [[Module:Message box/configuration]] [20:18:55]

https://meta.miraheze.org/wiki/Module:Message_box/configuration [20:18:56] [ Module:Message box/configuration - Miraheze Meta ] - meta.miraheze.org [20:31:48] https://phabricator.miraheze.org/tag/redis-jobrunner/ -- new project for the new software :P [20:31:49] [ Redis-JobRunner · Workboard ] - phabricator.miraheze.org [20:32:02] *new as in 'for us maintaining it' [20:34:58] yeah [20:37:29] Reception123: secondary thought, I'm not going to create a task yet as I'd like to do some more work on planning before that point [20:42:21] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.91, 1.50, 1.20 [20:46:17] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.23, 1.46, 1.26 [20:51:11] JohnLewis, ```Exception experienced creating the wiki. Error is: Redis server error: socket error on read socket``` [20:51:45] https://meta.miraheze.org/wiki/Special:RequestWikiQueue/17614#mw-section-comments [20:51:45] ^ Here's the wiki request [20:51:46] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [20:53:32] I just noticed the reason box on Special:RequestWiki is only the height of one line instead of like, a paragraph-sized box [20:53:48] I bet people would write more descriptive reasons if the box had more space [20:54:44] You can write more in it, but I don't think validation can be added to textarea in the same way so it wouldn't be a required field. Not sure though. [20:55:04] dorito, yeah, perhaps we could explore that [20:55:11] I know you can write more in it, and I'm not talking about validation or anything I just mean the size of the box itself [20:56:06] I know, not possible with mediawiki htmlForm, I don't think. [20:56:25] dorito, I think Universal_Omega is just saying that the paragraph-sized textarea box may not allow validation so that is something we'd have consider as we do want to have validation in that field [20:56:26]