[00:02:31] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.93, 1.54, 1.26 [00:04:30] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.59, 1.16, 1.15 [00:39:55] JohnLewis: I have made some changes to the job queue dashboard, I have added a job type tickbox that's a) automatically updated (when a new job is added, as long as there's a redis key for the new job) and b) allows you to choose your preferred set of job types for detailed information [00:40:09] if you don't like it, you can restore your own version via https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?editview=versions&orgId=1&var-jobtype=All [00:40:10] [ Grafana ] - grafana.miraheze.org [00:42:18] Looks like the LocalGlobalUserPageCacheUpdate jobs spiked again :P [00:42:46] night [00:43:08] SPF|Cloud, your version looks good as well, any way we could dark the scroll bar slider? I find it hard to find in order to scroll [00:43:27] Would be very challenging for users with visual impairments [01:15:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYAAZ [01:15:12] [02miraheze/services] 07MirahezeSSLBot 03cb6f505 - BOT: Updating services config for wikis [03:23:51] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.89, 1.37, 1.03 [03:25:48] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.72, 1.10, 0.98 [03:37:39] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.32, 1.81, 1.30 [03:39:39] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.00, 1.52, 1.25 [05:20:23] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [06:27:50] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki crappypastawiki --username-prefix="wikia:lolpasta" /home/reception/lolpasta_pages_full.xml [06:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:33] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JYxws [06:36:34] [02miraheze/mw-config] 07RhinosF1 03b4711ed - deploy future of flagged revs to test3wiki [06:36:36] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [06:36:50] [02mw-config] 07RhinosF1 opened pull request 03#3818: deploy future of flagged revs to test3wiki - 13https://git.io/JYxwC [06:37:56] miraheze/mw-config - RhinosF1 the build passed. [06:41:42] [02mw-config] 07RhinosF1 closed pull request 03#3818: deploy future of flagged revs to test3wiki - 13https://git.io/JYxwC [06:41:44] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYxrf [06:41:46] [02miraheze/mw-config] 07RhinosF1 0366724f9 - deploy future of flagged revs to test3wiki (#3818) [06:41:50] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [06:41:52] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [06:42:46] miraheze/mw-config - RhinosF1 the build passed. [06:43:09] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-4 [+0/-0/±1] 13https://git.io/JYxrO [06:43:10] [02miraheze/mw-config] 07Reception123 03e32e9da - add conflict for myvariables and approvedrevs [06:43:12] [02mw-config] 07Reception123 created branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [06:43:13] [02mw-config] 07Reception123 opened pull request 03#3819: add conflict for myvariables and approvedrevs - 13https://git.io/JYxr3 [06:44:31] miraheze/mw-config - Reception123 the build passed. [06:55:11] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-3818-RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JYxod [06:55:13] [02miraheze/mw-config] 07RhinosF1 037cc1924 - Revert "deploy future of flagged revs to test3wiki (#3818)" [06:55:14] [02mw-config] 07RhinosF1 created branch 03revert-3818-RhinosF1-patch-2 - 13https://git.io/vbvb3 [06:55:16] [02mw-config] 07RhinosF1 opened pull request 03#3820: Revert "deploy future of flagged revs to test3wiki" - 13https://git.io/JYxoN [06:55:38] [02mw-config] 07RhinosF1 closed pull request 03#3820: Revert "deploy future of flagged revs to test3wiki" - 13https://git.io/JYxoN [06:55:39] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYxoA [06:55:41] [02miraheze/mw-config] 07RhinosF1 03f7b9bc4 - Revert "deploy future of flagged revs to test3wiki (#3818)" (#3820) [06:56:09] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: rhinosf1, last run 13 minutes ago with 0 failures [06:56:26] miraheze/mw-config - RhinosF1 the build passed. [06:56:44] miraheze/mw-config - RhinosF1 the build passed. [06:58:09] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:11:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 6.21, 5.29 [07:13:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.55, 5.44, 5.12 [07:14:15] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-5 [+0/-0/±1] 13https://git.io/JYxPq [07:14:17] [02miraheze/mw-config] 07Reception123 034bebc11 - approvedrevs and myvariables conflict sitenotice [07:14:18] [02mw-config] 07Reception123 created branch 03Reception123-patch-5 - 13https://git.io/vbvb3 [07:14:20] [02mw-config] 07Reception123 opened pull request 03#3821: approvedrevs and myvariables conflict sitenotice - 13https://git.io/JYxPm [07:15:21] miraheze/mw-config - Reception123 the build passed. [07:17:06] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-5 [+0/-0/±1] 13https://git.io/JYxPy [07:17:07] [02miraheze/mw-config] 07Reception123 039620803 - Update Sitenotice.php [07:17:09] [02mw-config] 07Reception123 synchronize pull request 03#3821: approvedrevs and myvariables conflict sitenotice - 13https://git.io/JYxPm [07:17:17] [02mw-config] 07Reception123 closed pull request 03#3819: add conflict for myvariables and approvedrevs - 13https://git.io/JYxr3 [07:17:19] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYxP9 [07:17:20] [02miraheze/mw-config] 07Reception123 03e450a42 - add conflict for myvariables and approvedrevs (#3819) [07:17:22] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-4 - 13https://git.io/vbvb3 [07:17:23] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-4 [07:18:06] miraheze/mw-config - Reception123 the build passed. [07:18:21] miraheze/mw-config - Reception123 the build passed. [07:18:22] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.01, 7.57, 5.93 [07:19:19] [02mw-config] 07Reception123 closed pull request 03#3821: approvedrevs and myvariables conflict sitenotice - 13https://git.io/JYxPm [07:19:20] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYxXe [07:19:22] [02miraheze/mw-config] 07Reception123 03594d96e - approvedrevs and myvariables conflict sitenotice (#3821) [07:19:23] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-5 [07:19:25] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-5 - 13https://git.io/vbvb3 [07:20:21] miraheze/mw-config - Reception123 the build passed. [07:20:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.93, 6.53, 5.75 [07:24:38] JohnLewis: paladox https://phabricator.miraheze.org/T7112 [07:24:39] [ ⚓ T7112 JobQueueError from line 778 of /srv/mediawiki/w/includes/jobqueue/JobQueueRedis.php: Redis server error: socket error on read socket ] - phabricator.miraheze.org [07:36:10] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [08:18:02] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JYxQ7 [08:18:04] [02miraheze/ssl] 07Reception123 036a2b6b0 - add www.winenjoy.net cert [08:23:13] !log MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_user = '211266' AND euw_wiki = 'd45t5g8r1g45rg4r54r5t1wiki'; [08:23:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:21] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [08:45:59] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [10:02:14] RECOVERY - stablestate.org - reverse DNS on sslhost is OK: rDNS OK - stablestate.org reverse DNS resolves to cp10.miraheze.org [10:05:47] Reception123: if you’re running importDump, how is it the jobqueue software on jobrunner3? [10:18:23] whoops, my bad [10:18:57] but yeah, we do need to do something as it's also been reported about CW yesterday [10:20:23] Tasks to solve it were created yesterday [10:27:13] JohnLewis: what's best way to deploy a seperate branch for config once 1.36 moves to test3wiki? Will we have to add it to puppet like we do for MediaWiki repo? [10:27:13] RhinosF1: 2021-04-08 - 00:51:41BST tell RhinosF1 see Lake's message, ` has Score been fixed?`, at 22:32 your time [10:27:25] @Lake: no [10:28:39] If you want auto updating, yes. If you want manual then just git checkout [10:30:38] JohnLewis: what about during the deploy? I'm expecting we might see every FR wiki fatal. Would we just merge the config change after puppet is stopped and then it'll go out when puppet runs? [10:31:34] Config during a deploy shouldn't go live to the world immediately [10:32:18] So yes just merge after puppet is stopped everywhere [10:32:47] We already have $branch_mw_config in puppet so \o/ for test3 [11:26:50] [02miraheze/jobrunner-service] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYpsU [11:26:51] [02miraheze/jobrunner-service] 07JohnFLewis 03de7d72b - Remove rootjobs regularly [11:27:28] Reception123: ^ should be a fix for the connection errors at least temporarily. I need more time to investigate more effective alternatives [11:30:11] JohnLewis: what's the 3k jobs hanging round? [11:30:38] de-duplicated GlobalUserPage jobs [11:32:08] Why do they hang round though? They never run so shouldn't they be dropped [11:39:41] JohnLewis: thanks! [11:39:54] I'll try to run agajn [11:39:56] *Again [11:49:26] They never run, so never get dropped [11:49:35] it’s a bug it seems [12:00:05] working fine now [12:04:20] !log renamed greatcharacterswikiwiki to greatcharacterswiki [12:04:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:34] !log reception@jobrunner3:/mnt/mediawiki-static$ sudo -u www-data mv /mnt/mediawiki-static/greatcharacterswikiwiki /mnt/mediawiki-static/greatcharacterswiki [12:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:30] RhinosF1: FYI https://phabricator.miraheze.org/T7113 [12:06:31] [ ⚓ T7113 Issues with rename script ] - phabricator.miraheze.org [12:08:41] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 260s [12:13:41] I can't actually see in the script where the moving directories in /mnt/mediawiki-static is done though, am I missing something? [12:17:26] Reception123: ack [12:17:59] There's a fix matomo script in my homedir [12:18:07] I think some stuff are in magic repo [12:18:07] ok [12:18:33] RhinosF1: you mean for /mnt/mediawiki-static? I'm a bit confused as to how that process takes place I can't seem to find it [12:19:01] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYpl5 [12:19:02] [02miraheze/MatomoAnalytics] 07paladox 03717fa53 - Fix undefined variable [12:19:04] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fN4LT [12:19:05] [02MatomoAnalytics] 07paladox opened pull request 03#48: Fix undefined variable - 13https://git.io/JYplF [12:20:01] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYp8k [12:20:02] [02miraheze/MatomoAnalytics] 07paladox 036b54bd3 - Update CHANGELOG [12:20:03] miraheze/MatomoAnalytics - paladox the build passed. [12:20:04] [02MatomoAnalytics] 07paladox synchronize pull request 03#48: Fix undefined variable - 13https://git.io/JYplF [12:20:24] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYp8G [12:20:26] [02miraheze/MatomoAnalytics] 07paladox 03e6194af - Update extension.json [12:20:27] [02MatomoAnalytics] 07paladox synchronize pull request 03#48: Fix undefined variable - 13https://git.io/JYplF [12:20:35] paladox: thanks :) would the error there also be causing the issue with moving the dirs in static though? [12:21:05] miraheze/MatomoAnalytics - paladox the build passed. [12:21:27] miraheze/MatomoAnalytics - paladox the build passed. [12:21:50] [02MatomoAnalytics] 07paladox closed pull request 03#48: Fix undefined variable - 13https://git.io/JYplF [12:21:51] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JYp80 [12:21:53] [02miraheze/MatomoAnalytics] 07paladox 031d6d1f2 - Fix undefined variable (#48) [12:21:54] [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 [12:21:56] [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fN4LT [12:22:50] miraheze/MatomoAnalytics - paladox the build passed. [12:24:25] Reception123: https://github.com/miraheze/MirahezeMagic/blob/master/includes/MirahezeMagicHooks.php#L69 [12:24:26] [ MirahezeMagic/MirahezeMagicHooks.php at master · miraheze/MirahezeMagic · GitHub ] - github.com [12:25:00] Reception123: possibly, yeh [12:25:04] Someone needs to deploy it [12:26:50] paladox: could firejail also impact given it's using shell:: [12:39:19] will deploy [12:41:01] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYpRM [12:41:02] [02miraheze/mediawiki] 07Reception123 0388da7de - update MatomoAnalytics [12:59:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JYpzJ [12:59:12] [02miraheze/MirahezeMagic] 07translatewiki 0384913cb - Localisation updates from https://translatewiki.net. [12:59:13] [ Main page - translatewiki.net ] - translatewiki.net [13:00:11] miraheze/MirahezeMagic - translatewiki the build passed. [13:05:24] RhinosF1 yes, that's possible [13:11:50] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JYpgb [13:11:52] [02miraheze/MirahezeMagic] 07paladox 03667427a - Do not use firejail for Shell::Commands [13:11:53] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX [13:11:55] [02MirahezeMagic] 07paladox opened pull request 03#244: Do not use firejail for Shell::Commands - 13https://git.io/JYpgN [13:11:57] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:11:59] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:12:34] RhinosF1 [13:12:44] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:12:44] ack [13:12:47] Reception123: ^ [13:12:51] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:12:53] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [13:12:57] miraheze/MirahezeMagic - paladox the build passed. [13:18:05] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JYpae [13:18:07] [02miraheze/MirahezeMagic] 07paladox 03ec9c757 - Update generateMirahezeSitemap.php [13:18:08] [02MirahezeMagic] 07paladox synchronize pull request 03#244: Do not use firejail for Shell::Commands - 13https://git.io/JYpgN [13:19:12] miraheze/MirahezeMagic - paladox the build passed. [13:19:50] [02MirahezeMagic] 07paladox closed pull request 03#244: Do not use firejail for Shell::Commands - 13https://git.io/JYpgN [13:19:51] [02miraheze/MirahezeMagic] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/JYpa4 [13:19:53] [02miraheze/MirahezeMagic] 07paladox 03383f69e - Merge pull request #244 from miraheze/paladox-patch-2 [13:19:54] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 [13:19:56] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX [13:20:49] miraheze/MirahezeMagic - paladox the build passed. [13:21:13] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYpaK [13:21:15] [02miraheze/mediawiki] 07paladox 03795aacd - Update MirahezeMagic [13:32:43] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 193s [13:34:42] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:34:45] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:49] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:35:04] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:35:54] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:56] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:05:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.51, 6.34, 4.81 [14:05:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 6.28, 5.12 [14:06:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 6.62, 4.98 [14:07:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 6.90, 5.22 [14:07:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.46, 6.81, 5.44 [14:08:12] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.60, 6.39, 5.08 [14:09:56] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.78, 6.88, 5.65 [14:11:15] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.20, 6.30, 5.38 [14:11:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.07, 6.57, 5.68 [14:27:02] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [15:19:34] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pt.graalmilitary.com' expires in 15 day(s) (Sat 24 Apr 2021 15:13:52 GMT +0000). [15:22:02] PROBLEM - egoish.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'egoish.org' expires in 15 day(s) (Sat 24 Apr 2021 15:16:03 GMT +0000). [15:26:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYpNF [15:26:19] [02miraheze/ssl] 07MirahezeSSLBot 0333cec65 - Bot: Update SSL cert for pt.graalmilitary.com [15:29:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYpAw [15:29:27] [02miraheze/ssl] 07MirahezeSSLBot 030aa160e - Bot: Update SSL cert for egoish.org [16:03:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.67, 5.76, 4.48 [16:05:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.20, 6.67, 4.95 [16:07:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.09, 6.29, 5.02 [16:07:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.98, 6.12, 5.01 [17:30:35] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 2.64, 6.51, 3.54 [17:31:02] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [17:34:01] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [17:34:03] PROBLEM - wiki.umbrel.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.umbrel.tech could not be found [17:34:34] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.58, 3.12, 2.81 [17:40:55] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: rDNS OK - warmage.necrocity.co.uk reverse DNS resolves to cp10.miraheze.org [17:41:05] RECOVERY - wiki.umbrel.tech - reverse DNS on sslhost is OK: rDNS OK - wiki.umbrel.tech reverse DNS resolves to cp11.miraheze.org [19:08:46] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.83, 1.68, 1.17 [19:10:45] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.78, 1.41, 1.14 [19:48:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JYhat [19:48:02] [02miraheze/puppet] 07paladox 0374c4bac - openldap: Add support for monitoring [19:48:04] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:48:05] [02puppet] 07paladox opened pull request 03#1734: openldap: Add support for monitoring - 13https://git.io/JYhaq [19:51:02] 1.35.13 and 1.35.2 is out 👀 https://lists.wikimedia.org/pipermail/wikitech-l/2021-April/094418.html [19:51:02] [ [Wikitech-l] Security and maintenance release: 1.31.13 / 1.35.2 ] - lists.wikimedia.org [19:51:30] @doritio: we're looking at it now [19:51:45] 👍 [19:52:31] There's no tag yet on git [20:33:58] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.34, 5.88, 4.83 [20:34:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.11, 6.89, 5.53 [20:35:59] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.48, 6.56, 5.19 [20:36:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.04, 6.79, 5.65 [20:37:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.67, 6.07, 5.16 [20:38:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.73, 6.10, 5.53 [20:43:00] https://gerrit.wikimedia.org/r/q/project:mediawiki%252Fcore+branch:REL1_36 [20:43:01] heh [20:56:15] paladox: yeah I plan to do our cut in the morning if everything is there [21:03:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYhPy [21:03:37] [02miraheze/mw-config] 07paladox 0367c7b16 - Set $wgInternalServer and replace url with http:// [21:03:38] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [21:03:40] [02mw-config] 07paladox opened pull request 03#3822: Set $wgInternalServer and replace url with http:// - 13https://git.io/JYhP9 [21:04:40] miraheze/mw-config - paladox the build passed. [21:05:43] [02mw-config] 07paladox closed pull request 03#3822: Set $wgInternalServer and replace url with http:// - 13https://git.io/JYhP9 [21:05:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYhXT [21:05:46] [02miraheze/mw-config] 07paladox 03ada0414 - Set $wgInternalServer and replace url with http:// (#3822) [21:05:48] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [21:05:49] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [21:06:45] miraheze/mw-config - paladox the build passed. [21:09:15] [02miraheze/mediawiki] 07paladox pushed 03172 commits to 03REL1_35 [+8/-0/±497] 13https://git.io/JYhX8 [21:09:17] [02miraheze/mediawiki] 07reedy 0316ec13f - Start 1.35.2 [21:09:18] [02miraheze/mediawiki] 07legoktm 03e6f56f7 - Update git submodules [21:09:20] [02miraheze/mediawiki] 07jdforrester 038f59d99 - User: Deprecate confusingly-named isLoggedIn() method [21:09:21] [02miraheze/mediawiki] ... and 169 more commits. [21:10:02] [02mediawiki] 07dependabot[bot] edited pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYSbP [21:10:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYhXo [21:10:59] [02mediawiki] 07dependabot[bot] edited pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYSbP [21:11:00] [02mediawiki] 07dependabot[bot] closed pull request 03#1423: Bump extensions/VisualEditor from `be34a2d` to `dc85118` - 13https://git.io/JYSbP [21:11:02] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/VisualEditor-dc85118 - 13https://git.io/vbL5b [21:11:03] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/VisualEditor-dc85118 [21:11:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.32, 5.92, 4.82 [21:12:49] PROBLEM - jobrunner4 Puppet on jobrunner4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [21:13:05] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [21:13:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.80, 5.12, 4.67 [21:13:55] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [21:13:56] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [21:14:34] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [21:14:44] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [21:15:09] paladox: please purge the powered by mediawiki logos too [21:16:22] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.040 second response time [21:22:23] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20764 bytes in 0.316 second response time [21:29:19] [6199ea5636aa19b1ffacb0e6] [no req] Error from line 184 of /srv/mediawiki/w/vendor/wikimedia/object-factory/src/ObjectFactory.php: Class 'MediaWiki\Logger\Monolog\SyslogHandler' not found [21:29:20] uh [21:30:24] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 868 bytes in 0.068 second response time [21:31:10] paladox: composer needs ran I think? (Also vendor submodule needs updated, if I'm not mistaken?) [21:31:19] i've run composer [21:31:28] and vendor's been updated [21:32:19] Oh, hmm then, paladox: did you use composer version 1 or 2? I think composer V2 is used on 1.35.2, or is it just that it's now supported? I'm not really sure, just ideas. [21:32:25] composer 1 [21:32:44] 1.35.2 should support 1 and 2, though saying that i did try 2 but that failed. [21:32:56] also the class it wants is inside mediawiki core not in a submodule [21:33:14] Oh. [21:33:39] The only thing I know is I got that exact error locally before running composer but not after. [21:34:00] oh, hold on [21:34:22] Universal_Omega: it should be okay but https://phabricator.miraheze.org/T7116 [21:34:23] [ ⚓ T7116 Consider installing SecureLinkFixer ] - phabricator.miraheze.org [21:35:35] ok moved forward [21:36:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYhDz [21:36:15] [02miraheze/mw-config] 07paladox 03bab892f - Set call to LogstashFormatter::__construct to pass '' for [21:36:23] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20764 bytes in 0.333 second response time [21:36:33] works! [21:36:51] had to do composer update [21:37:22] miraheze/mw-config - paladox the build passed. [21:38:28] !log depool mw8 [21:38:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:40:54] PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: 1 backends are down. mw8 [21:41:04] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp10.miraheze.org [21:41:23] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 1 backends are down. mw8 [21:42:00] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8 [21:42:00] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [21:45:59] !log rebuild lc on mw8 [21:46:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:05] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:48:49] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:50:30] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:50:58] !log repool mw8 [21:51:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:15] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.88, 6.51, 5.15 [21:51:20] RECOVERY - cp10 Varnish Backends on cp10 is OK: All 7 backends are healthy [21:51:56] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.42, 7.13, 5.51 [21:51:58] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [21:52:00] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [21:52:54] RECOVERY - cp11 Varnish Backends on cp11 is OK: All 7 backends are healthy [21:53:16] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.59, 5.42, 4.93 [21:53:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.00, 5.87, 5.24 [21:55:36] !log rebuild lc on jobrunner[34] [21:55:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:54] !log depool mw9 [21:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:00:54] PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: 1 backends are down. mw9 [22:01:19] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 1 backends are down. mw9 [22:01:57] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.90, 6.40, 5.55 [22:01:59] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw9 [22:02:00] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw9 [22:02:32] Universal_Omega: https://github.com/Universal-Omega/DynamicPageList3 doesn't have rel1_36 branch [22:02:33] [ GitHub - Universal-Omega/DynamicPageList3 ] - github.com [22:03:00] RhinosF1: I know, ill do that today. [22:03:45] RhinosF1: done [22:06:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.41, 6.84, 5.72 [22:07:43] [02mediawiki] 07RhinosF1 created branch 03REL1_36 - 13https://git.io/vbL5b [22:07:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 868 bytes in 0.051 second response time [22:08:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.38, 6.06 [22:08:29] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-1/±0] 13https://git.io/JYh95 [22:08:31] [02miraheze/mediawiki] 07RhinosF1 037a69ab3 - reapply custom miraheze changes [22:08:32] [02mediawiki] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbL5b [22:08:40] [02mediawiki] 07RhinosF1 opened pull request 03#1446: reapply custom miraheze changes - 13https://git.io/JYh9F [22:08:56] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-1/±0] 13https://git.io/JYh9N [22:08:58] [02miraheze/mediawiki] 07RhinosF1 03852a9aa - Delete .gitignore [22:08:59] [02mediawiki] 07RhinosF1 synchronize pull request 03#1446: reapply custom miraheze changes - 13https://git.io/JYh9F [22:09:57] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JYh9h [22:09:57] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.34, 8.05, 6.67 [22:09:58] [02miraheze/mediawiki] 07RhinosF1 036273044 - Update composer.json [22:10:00] [02mediawiki] 07RhinosF1 synchronize pull request 03#1446: reapply custom miraheze changes - 13https://git.io/JYh9F [22:11:04] !log rebuild lc on mw9 [22:11:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:43] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20723 bytes in 0.172 second response time [22:15:18] RECOVERY - cp10 Varnish Backends on cp10 is OK: All 7 backends are healthy [22:15:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.27, 7.43, 6.94 [22:15:58] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [22:16:01] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [22:16:15] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JYhHz [22:16:17] [02miraheze/mediawiki] 07RhinosF1 0324a891d - Update CdnCacheUpdate.php [22:16:18] [02mediawiki] 07RhinosF1 synchronize pull request 03#1446: reapply custom miraheze changes - 13https://git.io/JYh9F [22:16:34] !log repool mw9 [22:16:37] !log depool mw10 [22:16:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:16:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:47] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.42, 6.78, 6.76 [22:17:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 1.93, 6.25, 6.67 [22:17:56] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:17:57] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JYhHP [22:17:59] [02miraheze/mediawiki] 07RhinosF1 03058ddbe - Update tables.sql [22:18:00] [02mediawiki] 07RhinosF1 synchronize pull request 03#1446: reapply custom miraheze changes - 13https://git.io/JYh9F [22:18:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.55, 7.85, 5.67 [22:19:17] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 1 backends are down. mw10 [22:20:01] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw10 [22:20:04] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw10 [22:20:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.78, 7.39, 4.26 [22:20:30] [02mediawiki] 07RhinosF1 edited pull request 03#1446: reapply custom miraheze changes - 13https://git.io/JYh9F [22:20:57] paladox: requested you and SPF|Cloud's review [22:22:01] ok, it looks good to me. Feel free to merge. [22:22:45] will wait on SPF|Cloud, no rush as I got to find sql patches [22:23:26] hi [22:23:29] what happened to mw10? [22:23:41] SPF|Cloud: 1.35.2 update [22:23:55] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:24:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.29, 7.67, 6.41 [22:24:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.05, 7.24, 4.95 [22:24:42] an update causes a depool? [22:25:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.020 second response time [22:25:24] SPF|Cloud: it requires composer changes [22:25:24] was this server depooled on cache proxies? [22:25:40] paladox is doing so [22:25:54] SPF|Cloud yes [22:26:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.59, 6.62, 6.18 [22:26:02] wait, that's right, icinga sends a critical even if depooled [22:26:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.79, 5.06 [22:26:18] nevermind [22:27:45] SPF|Cloud: can you review your hacks for 1.36 [22:28:05] if it's a 'no rush' thing, I will do it later [22:28:21] !log rebuild lc on mw10 [22:28:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:38] forgot that I had irccloud open, got disturbed while doing RL work [22:28:45] that's my fault :) [22:29:09] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20727 bytes in 0.180 second response time [22:29:42] afk [22:31:57] [02mediawiki] 07RhinosF1 edited pull request 03#1444: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMt [22:32:04] [02mediawiki] 07RhinosF1 edited pull request 03#1444: Dependabot: Switch to REL1_36 - 13https://git.io/JYbMt [22:32:05] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.76, 7.54, 5.95 [22:34:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.99, 8.07, 6.77 [22:34:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.59, 8.99, 6.66 [22:36:17] !log repool mw10 [22:36:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:22] !log depool mw11 [22:36:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:36:43]