[02:03:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.63, 6.53, 5.12 [02:04:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.14, 7.61, 5.44 [02:05:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.97, 8.13, 5.99 [02:05:28] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.82, 5.55, 4.94 [02:07:13] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.60, 7.87, 6.15 [02:08:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.81, 6.64, 5.62 [02:11:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.54, 6.75, 6.10 [02:15:05] [02CreateWiki] 07Universal-Omega opened pull request 03#211: [BUG FIX] fix logging when wiki is not private - 13https://git.io/JORFO [02:16:00] miraheze/CreateWiki - Universal-Omega the build passed. [02:25:12] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [02:35:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.94, 6.24, 5.36 [02:37:19] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.35, 5.67, 5.25 [03:01:17] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: connect to address iceria.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:02:24] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.iceria.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:55:29] PROBLEM - iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iceria.org could not be found [03:59:58] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [04:18:43] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 294s [04:26:40] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.92, 4.05, 2.58 [04:30:30] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.91, 3.70, 2.77 [04:37:43] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.78, 1.53, 1.09 [04:39:43] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.18, 1.36, 1.08 [04:43:57] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.65, 3.35, 3.17 [05:05:12] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.99, 3.41, 3.27 [05:06:29] [02CreateWiki] 07Reception123 closed pull request 03#211: [BUG FIX] fix logging when wiki is not private - 13https://git.io/JORFO [05:06:31] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JO0Gm [05:06:32] [02miraheze/CreateWiki] 07Universal-Omega 0390d6266 - [BUG FIX] fix logging when wiki is not private (#211) [05:07:07] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.04, 3.62, 3.36 [05:07:32] miraheze/CreateWiki - Reception123 the build passed. [05:09:02] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.81, 3.65, 3.40 [05:16:43] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.63, 3.33, 3.40 [05:26:46] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JO0cY [05:26:48] [02miraheze/mediawiki] 07Reception123 03a648d36 - Update CW [05:35:58] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.27, 3.61, 3.41 [05:39:49] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.42, 3.32, 3.36 [05:41:07] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:42:25] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:42:39] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [05:43:49] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.85, 3.58, 3.46 [05:45:49] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 3.32, 3.40, 3.40 [06:04:31] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:04:38] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:05:08] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:57:01] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [08:09:20] [02CreateWiki] 07Universal-Omega opened pull request 03#212: [BUG FIX] fix potential "deprecated" notice - 13https://git.io/JO0i1 [08:10:17] miraheze/CreateWiki - Universal-Omega the build passed. [08:55:55] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.61, 5.15, 2.80 [08:57:55] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.89, 3.54, 2.49 [08:59:54] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.28, 2.43, 2.21 [09:38:28] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.63, 3.75, 2.14 [09:40:29] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.68, 2.60, 1.91 [10:04:28] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.87, 3.70, 2.19 [10:06:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.11, 2.79, 2.04 [10:16:11] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [10:39:15] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 230s [10:41:13] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 197s [10:43:13] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [11:11:56] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.88, 3.63, 2.02 [11:13:54] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.62, 2.55, 1.81 [12:47:15] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.73, 4.75, 2.93 [12:49:12] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.34, 3.78, 2.79 [12:51:10] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.02, 2.77, 2.53 [13:11:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.50, 8.33, 5.22 [13:17:46] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.31, 3.14, 3.79 [13:19:43] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.33, 2.18, 3.36 [13:36:16] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Mon 07 Jun 2021 22:52:26 GMT +0000. [13:37:23] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Mon 07 Jun 2021 22:52:26 GMT +0000. [13:42:18] RECOVERY - iceria.org - reverse DNS on sslhost is OK: rDNS OK - iceria.org reverse DNS resolves to cp10.miraheze.org [13:49:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JOE0r [13:49:06] [02miraheze/mediawiki] 07paladox 031847836 - Update Metrolook [13:53:00] RECOVERY - www.iceria.org - reverse DNS on sslhost is OK: rDNS OK - www.iceria.org reverse DNS resolves to cp11.miraheze.org [13:57:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.49, 6.56, 5.17 [13:59:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.47, 6.53, 5.34 [14:01:55] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 4.40, 5.97, 3.20 [14:05:57] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.90, 3.07, 2.63 [14:11:54] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.82, 3.85, 2.65 [14:13:51] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.44, 3.18, 2.53 [14:17:44] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.85, 4.60, 3.26 [14:19:41] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.77, 3.37, 2.96 [14:25:33] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.27, 3.41, 3.26 [14:27:31] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.02, 2.57, 2.97 [15:14:35] @R4356th: if you do any adding of exts to mw repo, make sure it's added to 1.36 branch too pls :) [15:25:02] [02mw-config] 07Reception123 closed pull request 03#3837: Add extendedconfirmedprotected restriction level to famepediawiki - 13https://git.io/JOR4y [15:25:04] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOEDm [15:25:05] [02miraheze/mw-config] 07dmehus 035d19112 - Add extendedconfirmedprotected restriction level to famepediawiki (#3837) [15:26:05] miraheze/mw-config - Reception123 the build passed. [15:27:29] [02mw-config] 07dmehus commented on pull request 03#3838: Add editextendedconfirmedprotected restrict. level to documentcontrolwiki - 13https://git.io/JOED9 [15:34:23] [02mw-config] 07dmehus synchronize pull request 03#3838: Add editextendedconfirmedprotected restrict. level to documentcontrolwiki - 13https://git.io/JOR2q [15:34:53] [02mw-config] 07dmehus commented on pull request 03#3838: Add editextendedconfirmedprotected restrict. level to documentcontrolwiki - 13https://git.io/JOESE [15:35:23] miraheze/mw-config - dmehus the build passed. [15:36:32] Reception123, ^ I think I got it resolved [15:37:17] dmehus: yep [15:37:24] 👍 [15:37:33] dmehus: ah no, see your changes [15:37:39] hrm? [15:37:45] dmehus: you need to fix a tab [15:38:15] oh [15:38:41] ` 'editextendedconfirmedprotected',` needs inserting on line 3598? [15:40:26] [02mw-config] 07dmehus synchronize pull request 03#3838: Add editextendedconfirmedprotected restrict. level to documentcontrolwiki - 13https://git.io/JOR2q [15:40:36] no this is the problem [15:40:37] https://github.com/miraheze/mw-config/pull/3838/files#diff-35b93cff8d87232a32e7a37a41c1e96682b8c911f123f1ffec8f3291b5bbfdb0R2665 [15:40:37] [ Add editextendedconfirmedprotected restrict. level to documentcontrolwiki by dmehus · Pull Request #3838 · miraheze/mw-config · GitHub ] - github.com [15:40:39] it needs to be tabbed once [15:41:24] miraheze/mw-config - dmehus the build passed. [15:42:09] Reception123, oh well then there was two problems then. Let me look at that [15:42:39] dmehus: github should have gave you a page where you could have fixed it [15:43:10] [02mw-config] 07dmehus synchronize pull request 03#3838: Add editextendedconfirmedprotected restrict. level to documentcontrolwiki - 13https://git.io/JOR2q [15:44:07] miraheze/mw-config - dmehus the build passed. [15:45:04] RhinosF1, I did fix the merge conflict. The tab and missed line insertions were just errors in my copying and pasting [15:45:12] Reception123, ^ should be fixed now [15:46:40] dmehus: then you didn't fix it correctly. Copying and pasting shouldn't have been needed when fixing a merge conflict. [15:47:10] But I'd have just done one PR for the 2 wikis as git was never gonna merge that cleanly [15:48:54] RhinosF1, no I did, I simply copied the missed lines from the part below the `=====` to the part above the equal sign. I just missed copying two lines for the other PR and tabbing once. I think it was pretty straightforward. It was my first merge conflict I had to resolve, and I think I did fine :) [15:49:24] Right [15:49:25] I didn't have instructions on the best way to resolve it, so went with what seemed right to me. That's how we learn :) [15:49:47] When in the resolve, you just edit to what I should look like after your edit [15:50:01] Because git couldn't apply your change on top of the current file [15:50:09] As it had changed in the same place [15:53:46] RhinosF1, yeah, that's what I did... I just think this was an error on my part, not a Git/GitHub error, in that I didn't copy the right lines. I should've more closely compared the lines for both wikis [15:54:33] Yeah [15:54:45] You need to look carefully when merging sections [15:55:12] Git can't resolve conflicts if it affects above or the same section in the file sometimes [15:55:22] Same section for definite [15:55:26] Above can vary [16:05:21] dmehus: also your mass message is weird [16:06:09] RhinosF1, yeah, I've never seen that before. Any idea what caused that? [16:06:29] dmehus: no [16:06:42] If my theory is right it's a fairly easy fix [16:06:51] But why it happened I don't know [16:14:22] Universal_Omega: did you ever look at the core sqls for 1.36 [16:14:56] RhinosF1: not yet I forgot I'll do today sorry. [16:15:03] Np [16:15:08] [02CreateWiki] 07Reception123 closed pull request 03#212: [BUG FIX] fix potential "deprecated" notice - 13https://git.io/JO0i1 [16:15:10] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOEdH [16:15:11] [02miraheze/CreateWiki] 07Universal-Omega 03744719d - [BUG FIX] fix potential "deprecated" notice (#212) [16:15:24] Don't think it'll be Friday until rc0 [16:15:50] And then we got to find time to actually do it [16:16:07] miraheze/CreateWiki - Reception123 the build passed. [16:18:17] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [16:20:51] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.58, 3.58, 2.43 [16:22:50] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.66, 2.84, 2.29 [16:28:47] > If my theory is right it's a fairly easy fix [16:28:47] oh, what's your theory? [16:28:47] > But why it happened I don't know [16:28:47] RhinosF1, ah :( [16:45:44] [02mw-config] 07Universal-Omega opened pull request 03#3839: Convert SimpleTooltip to use ExtensionRegistry - 13https://git.io/JOEpL [16:46:10] dmehus: the actor id is wrong [16:46:22] [02mw-config] 07Universal-Omega synchronize pull request 03#3839: Convert SimpleTooltip to use ExtensionRegistry - 13https://git.io/JOEpL [16:46:41] miraheze/mw-config - Universal-Omega the build passed. [16:47:21] miraheze/mw-config - Universal-Omega the build passed. [17:11:26] RhinosF1, oh interesting. Actor ID for me, MediaWiki message delivery, or SHEIKH? If either of the former two, why weren't the others affected? If the latter one, howwhy? or that's what we don't know? [17:11:50] dmehus: MediaWiki Message Delivery [17:11:53] ah [17:11:58] hrm, interesting [17:12:02] And the others arent system accounts [17:12:10] For some reason it's decided to use 0 [17:12:19] When that's wrong [17:12:25] It has an actual actor I'd [17:12:34] JohnLewis: bacula was never actually resized was it? https://phabricator.miraheze.org/T7038#139368 [17:12:35] [ ⚓ T7038 Existing Server Resource Request for bacula2 ] - phabricator.miraheze.org [17:12:45] the last thing I see from SAL is [17:12:46] 22:58 Southparkfan: remove downtime from bacula2, upgrade issues (ticket opened with RamNode) [17:12:49] so then we just need to figure out why/how it's just a problem for that one diff, and not all the other ones [17:12:54] I just want to do the actual comparing of stuff [17:12:59] To be sure [17:13:03] ah [17:13:15] Probably a random fluke [17:13:25] Or race condition [17:13:55] do we have a script to fix that, or would this have to be fixed with an SQL table update? [17:15:05] Manual query [17:15:58] RhinosF1, ack, fun () [17:16:52] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.05, 3.74, 2.21 [17:17:40] Yeah [17:17:45] I'll need to be careful [17:18:11] yeah [17:18:40] Reception123: is that what all the fail emails were about [17:18:50] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.94, 3.40, 2.27 [17:19:38] RhinosF1: I think so, from what I understand it should've been resized so we can have proper backups but there was an issue with RN [17:20:24] Ye [17:20:45] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Reception123, last run 14 minutes ago with 0 failures [17:22:45] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:24:23] random question: is the Puppet run schedule the same as for the other mw servers, or does it run on a different schedule and/or interval? [17:24:42] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±2] 13https://git.io/JOuf5 [17:24:43] [02miraheze/mediawiki] 07Reception123 03f789776 - Switch SimpleTooltip to Universal-Omega fork [17:24:50] s/schedule/schedule for test3wiki [17:24:50] dmehus meant to say: random question: is the Puppet run schedule for test3wiki the same as for the other mw servers, or does it run on a different schedule and/or interval? [17:24:56] dmehus: it's the same, otherwise that would mess things up [17:25:02] dmehus: oh, you mean only for test3? [17:25:14] also the same [17:25:18] you can see that here https://github.com/miraheze/puppet/blob/master/hieradata/hosts/test3.yaml [17:25:19] Reception123, ah, makes sense [17:25:19] [ puppet/test3.yaml at master · miraheze/puppet · GitHub ] - github.com [17:25:25] https://github.com/miraheze/puppet/blob/master/hieradata/hosts/test3.yaml#L12 [17:25:25] [ puppet/test3.yaml at master · miraheze/puppet · GitHub ] - github.com [17:25:28] cool, thanks :) [17:26:37] so if you wanted Puppet to be run only hourly, would you change `puppet_cron_time: '2,32'` to `puppet_cron_time: '2,30'`? [17:26:42] oops [17:26:47] s/2,30/2,0 [17:26:47] dmehus meant to say: so if you wanted Puppet to be run only hourly, would you change `puppet_cron_time: '2,32'` to `puppet_cron_time: '2,0'`? [17:29:03] No [17:29:16] Just 2 [17:30:03] [02mw-config] 07Reception123 closed pull request 03#3839: Convert SimpleTooltip to use ExtensionRegistry - 13https://git.io/JOEpL [17:30:04] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JOuJ2 [17:30:05] [02miraheze/mw-config] 07Universal-Omega 03d2951ce - Convert SimpleTooltip to use ExtensionRegistry (#3839) [17:30:32] RhinosF1, ack, ah wasn't sure if the second digit would be 0 or null. Makes sense :) [17:31:00] miraheze/mw-config - Reception123 the build passed. [17:31:19] Reception123: seems not [17:31:20] Reception123, any idea why your revisions on test3wiki aren't autochecked on test3wiki? `sysop` does have the `autoreview` user right [17:31:25] But it's a capacity issue with RN [17:31:49] JohnLewis: yeah, that's what it looked like. SPF|Cloud said he opened a RN ticket, I wonder if there's been any response [17:31:54] dmehus: that value is number of minutes past the hour [17:32:09] RhinosF1, ah, ack [17:32:15] dmehus: I don't know, many things have been done with that wiki so something is bound to be messed up [17:32:47] Reception123, yeah... we should consider deleting test3wiki and recreating it as a fresh wiki after the MW 1.36 upgrade is done [17:33:27] Honestly there's a bug upstream tbh about it sometimes happening [17:33:40] But flagged revs is slowly being made less messy [17:33:46] So just don't worry [17:33:56] Keep trying after each MW upgrade [17:36:08] yeah we could do that, it would be cleaner [17:38:17] RhinosF1, yeah I definitely prefer ApprovedRevs to FlaggedRevs for a variety of reasons [17:38:52] Reception123, yeah Omega did ask Paladox what he thought about resetting test3wiki and he did give his +1 to the idea [17:38:59] Probably [17:39:00] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:39:13] It's more the massive amount of extensions on there [17:39:19] yeah [17:39:20] Things are bound to conflict [17:39:27] definitely [17:39:29] yeah, the issue is all the tests and weird SQL stuff on there [17:39:30] It's not really a good testing setup tbh [17:39:37] 💯 [17:39:41] and of course for extensions, we can't have all enabled because of conflicts [17:39:45] it's not the ideal testing setup but it does the job [17:39:57] and remember there was a time when there was no testing setup at all [17:40:06] That upsets me [17:40:20] This is where the idea started - https://phabricator.miraheze.org/T2005 [17:40:21] [ ⚓ T2005 Purchase and configuration new testing / development server ] - phabricator.miraheze.org [17:40:21] ideally we should be testing extensions individually, but it is also good to test with other extensions as well, to discover conflicts, but yeah as far as whether an extension is compatible, fewer enabled extensions would be better [17:40:40] what upsets you RhinosF1? [17:40:50] Not having a test server [17:40:57] oh yeah, for two years [17:41:01] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:41:04] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:41:10] RhinosF1, oh [17:41:16] Ideally every extension would have full test coverage and we'd have an amazing CI & Beta Cluster [17:41:19] test1 began in late 2017 [17:41:23] And we'd be able to run full gates [17:41:24] was that why Paladox used to use TestWiki for testing? [17:41:40] is the 503 right now because of extension testing? [17:41:40] "full gates," what's that? [17:41:42] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:41:43] if I'm not mistaken I don't think pala dox was here yet when we didn't have test1 [17:42:00] @Lake: no [17:42:13] Getting a 500 and not only on my wiki [17:42:15] @Lake, there is a puppet failure on mw9, so could be because of that maybe? [17:42:34] dmehus: that changes to mediawiki would be tested fully and all deployments would trigger tests of every extension completely [17:42:39] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 43 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:42:47] @Kozd which wiki is it? [17:42:50] RhinosF1, oh [17:42:52] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 47 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:43:00] !log sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* [17:43:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:10] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki loginwiki [17:43:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:19] dmehus: that would make deployments take centuries though [17:43:26] true [17:43:37] And full test coverage just never happens [17:43:48] true [17:44:02] I'm getting random 503s and 500s [17:44:05] on all wikis [17:44:13] Reception123, could the 503 be related to Omega's PR you merged? [17:44:18] Yes, it definitely is [17:44:21] ah [17:44:35] it worked on test3 but I think the LC cache needs to finish rebuilding [17:44:36] dmehus: our own extensions have 0% coverage [17:44:46] Reception123, ah [17:45:18] RhinosF1, 0% coverage, you mean they're not installed outside of Miraheze? [17:45:37] dmehus: no there's no automated tests [17:45:45] Our extensions are only tested manually [17:45:48] And limited [17:45:51] Linted* [17:46:40] RhinosF1, ah, that's what you mean. yeah, makes sense [17:47:01] I really hope running LC fixes this otherwise we'll have to revert [17:47:04] but it has to because it worked on test3 [17:47:32] will we have to do an IR for this, since the errors lasted for more than a few minutes? [17:48:09] yeah, guess so [17:49:31] ah [17:57:17] !log cd /srv/mediawiki/w/extensions && sudo -u www-data git reset --hard origin/REL1_35 && sudo -u www-data git pull ; sudo -u www-data git submodule update on mw* [17:57:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:37] https://www.irccloud.com/pastebin/P5cbzHMC/ [17:58:37] [ Snippet | IRCCloud ] - www.irccloud.com [17:58:43] urgh, looking like a revert [17:59:09] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:59:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03revert-3839-patch-161 [+0/-0/±2] 13https://git.io/JOuI4 [17:59:12] [02miraheze/mw-config] 07Reception123 034121ea2 - Revert "Convert SimpleTooltip to use ExtensionRegistry (#3839)" [17:59:13] [02mw-config] 07Reception123 created branch 03revert-3839-patch-161 - 13https://git.io/vbvb3 [17:59:15] [02mw-config] 07Reception123 opened pull request 03#3840: Revert "Convert SimpleTooltip to use ExtensionRegistry" - 13https://git.io/JOuIB [18:00:10] [02mw-config] 07Reception123 closed pull request 03#3840: Revert "Convert SimpleTooltip to use ExtensionRegistry" - 13https://git.io/JOuIB [18:00:11] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JOuIw [18:00:13] [02miraheze/mw-config] 07Reception123 037c0ceae - Revert "Convert SimpleTooltip to use ExtensionRegistry (#3839)" (#3840) [18:00:13] miraheze/mw-config - Reception123 the build passed. [18:00:18] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±2] 13https://git.io/JOuIo [18:00:20] [02miraheze/mediawiki] 07Reception123 0361b574d - revert back to original SimpleTooltip (errors) [18:01:13] miraheze/mw-config - Reception123 the build passed. [18:01:48] dmehus: oh, so the PR was related to SimpleTooltip? [18:02:38] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:02:51] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:03:50] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:41] !log re-ran the scripts below [18:04:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:03] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:10] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:11:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.64, 18.41, 14.55 [18:12:30] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.45, 3.81, 2.02 [18:13:05] @Lake, yeah, it seems like re just need to rebuild the localisation cache afterward or something [18:13:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.56, 18.95, 15.26 [18:13:42] oh wait no I see Reception123 has reverted [18:13:52] so yeah seems like there was an issue with the commit [18:13:59] dmehus: yeah, git failed so I couldn't afford trying more things and keeping things down [18:14:10] Reception123, yeah, good call :) [18:14:20] next time I try this I'll instead disable puppet on mw* and do stuff on test3 like that to see if Puppet also fails there [18:14:28] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.54, 2.82, 1.88 [18:14:31] oh yeah, that's a good idea [18:17:56] not related, but if there's test3, is there a test1 and test2? [18:18:29] those were the old server names [18:18:37] so they just redirect to test3wiki [18:19:29] so as far as active servers, there's just testwiki (community testing; some sysadmin testing) and test3wiki (sysadmin testing) [18:19:37] @Lake yeah, every time a server is replaced even if it keeps the same function it gets a new number [18:19:44] so when test2 was replaced with a new server it became test3 [18:20:03] and that's why there's mw8/mw9/mw10/mw11, we don't actually have 11 MW servers [18:20:17] ooh I see [18:20:47] I mean, if I get millionaire someday, I will buy 11 servers for miraheze /s [18:21:27] Technically we actually only have three (or two?) cloud servers, and most of the VMs just reside on the cloud servers [18:22:10] yeah [18:22:26] a table is available here: https://meta.miraheze.org/wiki/Tech:Server_usage [18:22:27] [ Tech:Server usage - Miraheze Meta ] - meta.miraheze.org [18:22:37] + bacula + cp3, any others we keep separate? [18:22:41] * dmehus is looking [18:23:26] Reception123, could we add another column to that table at Tech:Server usage to identify which cloud server hosts which VMs? [18:23:33] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:31] yeah, we could sure [18:25:08] I could do it but I'd need the data in terms of which VMs are on which server. Is that data public anywhere (i.e., Grafana)? [18:27:00] oh yeah I meant to ask about that...we host a Zotero instance? That must be related to Citoid, I'm assuming, but that's pretty neat. [18:30:43] dmehus: the infoboxes have it [18:30:51] I can make it add a category [18:31:32] And yes zotero is for Citoid [18:33:38] RhinosF1, yeah, that's a good idea, actually, having the infobox add a category for which cloud server it's on [18:33:52] I might try and work on that unless you get to it first [18:35:22] PROBLEM - wiki.jacksonheights.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jacksonheights.nyc could not be found [18:35:28] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [18:35:44] https://meta.miraheze.org/w/index.php?title=Category:Servers_on_cloud1&action=edit&redlink=1 [18:35:45] [ Category:Servers on cloud1 - Miraheze Meta ] - meta.miraheze.org [18:35:49] https://meta.miraheze.org/w/index.php?title=Category:Servers_on_cloud2&action=edit&redlink=1 [18:35:50] [ Category:Servers on cloud2 - Miraheze Meta ] - meta.miraheze.org [18:35:53] https://meta.miraheze.org/w/index.php?title=Category:Servers_on_cloud3&action=edit&redlink=1 [18:35:55] [ Category:Servers on cloud3 - Miraheze Meta ] - meta.miraheze.org [18:36:36] dmehus: ^ [18:37:35] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Tue 29 Jun 2021 07:57:16 GMT +0000. [18:41:24] by the way, Meta should enable extension Mermaid if there's ever a need to make a diagram [18:41:29] it's really good [18:42:11] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp10.miraheze.org [18:42:19] RECOVERY - wiki.jacksonheights.nyc - reverse DNS on sslhost is OK: rDNS OK - wiki.jacksonheights.nyc reverse DNS resolves to cp10.miraheze.org [18:42:53] RhinosF1, thanks, and done [18:49:54] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 3.14, 4.01, 2.42 [18:51:09] RhinosF1, though I didn't see your redlinks so wasn't able to use them, but appreciate the thought :) [18:51:33] @Lake, oh interesting. I never looked into what Mermaid did [18:51:55] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.49, 3.90, 2.60 [18:52:27] We do have the Timeline extension, but the Mermaid extension sounds potentially useful on Meta yeah [18:53:05] RhinosF1, just to make sure, cloud3-5 are still running and cloud1-2 are decommissioned, right? [18:53:54] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 1.06, 2.88, 2.38 [18:54:21] dmehus: https://mermaid-js.github.io/mermaid-live-editor here's a live editor. It's pretty simple and there are different diagram/graphs [18:54:21] [ Mermaid live editor ] - mermaid-js.github.io [18:54:25] including timeline as well [18:55:28] @Lake, oh thanks [19:08:09] dmehus: yes [19:08:59] I think db* is cloud3 [19:09:06] But paladox will know [19:22:11] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [19:25:12] [02mw-config] 07Universal-Omega opened pull request 03#3841: Add writeapi to * when flow is enabled from ManageWikiExtensions - 13https://git.io/JOuZu [19:26:17] miraheze/mw-config - Universal-Omega the build passed. [19:27:14] [02mw-config] 07JohnFLewis commented on pull request 03#3841: Add writeapi to * when flow is enabled from ManageWikiExtensions - 13https://git.io/JOuZP [19:27:54] db* is cloud3 as it's the only SSD server [19:29:25] [02mw-config] 07Universal-Omega commented on pull request 03#3841: Add writeapi to * when flow is enabled from ManageWikiExtensions - 13https://git.io/JOuZj [19:36:05] RhinosF1, ah, thanks. :) [19:37:21] [02mw-config] 07dmehus commented on pull request 03#3841: Add writeapi to * when flow is enabled from ManageWikiExtensions - 13https://git.io/JOun1 [19:37:34] [02mw-config] 07dmehus edited a comment on pull request 03#3841: Add writeapi to * when flow is enabled from ManageWikiExtensions - 13https://git.io/JOun1 [19:49:55] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 3.86, 6.36, 3.56 [19:53:58] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.94, 3.64, 3.08 [19:55:57] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.59, 2.59, 2.76 [19:57:02] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.46, 2.54, 1.50 [19:59:05] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.18, 1.98, 1.42 [20:01:05]