[00:01:13] dmehus: ^ though, Ideally, we should create a new ManageWikiValidation helper class I think, with multiple options for validation and able to call from ManageWikiSettings, ManageWikiNamespaces. [00:01:54] Universal_Omega, ah, yeah, that sounds good. Is that difficult for you to do, to create a new MW Validation helper class? [00:02:23] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 6708 MB (17% inode=96%); [00:02:47] dmehus: yes, I personally don't know if I could, or even where I'd start honestly. Never done anything of the sort of validation needed, although I know it's possible. [00:03:40] But I sure can try/work on it. [00:04:36] [02mw-config] 07Universal-Omega commented on pull request 03#3773: Update wgHAWelcomeWelcomeUsername default - 13https://git.io/JqcYG [00:04:43] Universal_Omega, cool, that'd be awesome. For now, yeah, your fix seems great. [00:05:18] [02mw-config] 07dmehus commented on pull request 03#3773: Update wgHAWelcomeWelcomeUsername default - 13https://git.io/JqcYZ [00:05:48] [02mw-config] 07Universal-Omega closed pull request 03#3773: Update wgHAWelcomeWelcomeUsername default - 13https://git.io/JqcmH [00:05:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqcYc [00:05:51] [02miraheze/mw-config] 07Universal-Omega 036c78257 - Update wgHAWelcomeWelcomeUsername default (#3773) [00:05:52] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:05:54] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:06:50] miraheze/mw-config - Universal-Omega the build passed. [00:10:45] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.55, 1.74, 1.19 [00:12:40] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Fix echo notifications for declined requests - 13https://git.io/JqZ1z [00:12:44] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.78, 1.41, 1.14 [00:21:39] Unsure why that is a ‘fix’ when the removal was intentional and not a bug [00:25:19] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 13.48, 5.89, 2.73 [00:25:57] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Re-add echo notifications for declined requests - 13https://git.io/JqZ1z [00:26:34] yeah that's a better title [00:27:16] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Re-add echo notifications for declined requests - 13https://git.io/JqZ1z [00:27:47] JohnLewis: sorry, yes, I worded that poorly. [00:30:58] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Re-add echo notifications for declined requests - 13https://git.io/JqZ1z [00:31:18] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 0.36, 3.46, 2.92 [00:31:59] [02CreateWiki] 07Universal-Omega edited pull request 03#201: Re-add echo notifications for declined requests - 13https://git.io/JqZ1z [00:33:18] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.53, 2.48, 2.62 [00:37:36] ```[e804fbeb4de31d3727478c2a] 2021-03-12 00:35:36: Fatal exception of type "TypeError"``` [00:37:36] !sre when you get a minute to run a trace. Triggered when clicking through to paladox' test wiki creations (https://meta.miraheze.org/wiki/Special:RequestWikiQueue/...). Likely a minor, low priority fix, but worth investigating, I think [00:37:38] [ Internal error - Miraheze Meta ] - meta.miraheze.org [00:38:36] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jqc3c [00:38:38] [02miraheze/CreateWiki] 07Universal-Omega 03660851b - Fix formatting [00:38:39] [02CreateWiki] 07Universal-Omega synchronize pull request 03#201: Re-add echo notifications for declined requests - 13https://git.io/JqZ1z [00:39:41] miraheze/CreateWiki - Universal-Omega the build passed. [00:43:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.53, 3.79, 3.97 [00:43:34] dmehus: haven't run actually trace, but likely because there are no request IDs for them.... the logs link to `Special:RequestWikiQueue/...` [00:47:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.11, 3.94, 4.00 [00:58:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 246s [01:00:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 48s [01:20:55] Universal_Omega, yeah, likely related to the log link [01:30:44] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqcGs [01:30:46] [02miraheze/WikiDiscover] 07Universal-Omega 03fb6f685 - Add (any) option to sort options dropdowns and default to it [01:30:48] [02WikiDiscover] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [01:31:01] [02WikiDiscover] 07Universal-Omega opened pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcGG [01:32:03] miraheze/WikiDiscover - Universal-Omega the build passed. [01:32:59] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqcG4 [01:33:01] [02miraheze/WikiDiscover] 07Universal-Omega 03954cdc0 - adhere to 'any' option in filter dropdown [01:33:02] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcGG [01:33:58] miraheze/WikiDiscover - Universal-Omega the build passed. [01:35:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.49, 3.51, 3.99 [01:37:27] [02WikiDiscover] 07dmehus commented on pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcGa [01:38:30] [02WikiDiscover] 07Universal-Omega commented on pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcGP [01:43:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.38, 3.66, 3.84 [01:44:23] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqcGh [01:44:25] [02miraheze/WikiDiscover] 07Universal-Omega 03ae5c512 - remove unneeded code [01:44:26] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcGG [01:45:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.96, 3.75, 3.85 [01:45:19] [02WikiDiscover] 07Universal-Omega closed pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcGG [01:45:20] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqcGj [01:45:22] [02miraheze/WikiDiscover] 07Universal-Omega 035be503c - Add (any) option to sort options dropdowns and default to it (T6577) (#36) [01:45:23] [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:45:25] [02WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [01:45:25] miraheze/WikiDiscover - Universal-Omega the build passed. [01:46:17] miraheze/WikiDiscover - Universal-Omega the build passed. [01:48:06] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqcZk [01:48:07] [02miraheze/WikiDiscover] 07Universal-Omega 03052fa2a - put (any) on top for category filter dropdown [01:49:00] miraheze/WikiDiscover - Universal-Omega the build passed. [01:51:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.20, 3.81, 3.83 [01:55:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.69, 3.85, 3.86 [01:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.03, 3.72, 3.79 [02:05:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.54, 3.92, 3.91 [02:11:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.39, 4.10, 4.00 [02:26:21] [02WikiDiscover] 07dmehus commented on pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/Jqccm [02:40:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 286s [02:42:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 89s [03:40:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 190s [03:42:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:48:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 314s [04:02:23] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 3s [04:19:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.68, 3.49, 3.99 [04:21:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.37, 3.79, 4.03 [04:25:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.90, 3.85, 4.00 [04:27:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.09, 4.01, 4.05 [04:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.01, 3.56, 3.90 [04:55:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.31, 3.76, 3.89 [04:59:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.71, 3.98, 3.97 [05:01:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.00, 4.06, 4.01 [05:03:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.63, 3.92, 3.96 [05:07:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.59, 4.12, 4.03 [05:11:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.52, 3.90, 3.97 [05:13:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.09, 3.87, 3.94 [05:25:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.34, 3.82, 3.97 [05:27:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.19, 4.02, 4.03 [06:03:33] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqcRC [06:03:34] [02miraheze/landing] 07translatewiki 03ea2341e - Localisation updates from https://translatewiki.net. [06:03:35] [ Main page - translatewiki.net ] - translatewiki.net [06:03:36] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqcRW [06:03:37] [02miraheze/CreateWiki] 07translatewiki 03c81c1c1 - Localisation updates from https://translatewiki.net. [06:03:38] [ Main page - translatewiki.net ] - translatewiki.net [06:03:39] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JqcRl [06:03:40] [02miraheze/MirahezeMagic] 07translatewiki 03c474c1b - Localisation updates from https://translatewiki.net. [06:03:41] [ Main page - translatewiki.net ] - translatewiki.net [06:03:42] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JqcR8 [06:03:43] [02miraheze/ManageWiki] 07translatewiki 03caf7e76 - Localisation updates from https://translatewiki.net. [06:03:44] [ Main page - translatewiki.net ] - translatewiki.net [06:04:30] miraheze/CreateWiki - translatewiki the build passed. [06:04:31] miraheze/landing - translatewiki the build passed. [06:04:40] miraheze/MirahezeMagic - translatewiki the build passed. [06:04:49] miraheze/ManageWiki - translatewiki the build passed. [06:30:24] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 192s [06:32:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 257s [06:34:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 83s [06:43:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.49, 3.20, 3.86 [06:47:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.25, 3.65, 3.89 [06:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.58, 3.92, 3.99 [06:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 6.20, 4.54, 4.17 [07:15:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.30, 3.58, 3.98 [07:17:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.70, 3.84, 4.01 [07:22:53] PROBLEM - bacula2 Disk Space on bacula2 is WARNING: DISK WARNING - free space: / 103774 MB (10% inode=99%); [09:00:35] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.61, 3.18, 2.19 [09:02:36] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.15, 2.45, 2.04 [09:53:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.85, 3.80, 3.97 [09:55:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.41, 4.46, 4.20 [10:22:53] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 55605 MB (5% inode=99%); [10:27:19] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 6.77, 3.46, 1.57 [10:29:18] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.77, 2.78, 1.56 [10:30:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 207s [10:34:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 157s [10:35:30] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.19, 3.41, 2.43 [10:37:28] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.95, 3.28, 2.50 [10:38:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:55:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.29, 3.60, 4.00 [13:57:06] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.39, 3.90, 4.06 [14:11:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.67, 3.84, 3.99 [14:15:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.40, 3.99, 4.01 [14:17:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.73, 3.90, 3.98 [14:21:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.17, 3.95, 3.98 [14:43:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.37, 3.61, 3.91 [14:47:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.03, 3.56, 3.80 [14:51:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.72, 3.74, 3.82 [14:57:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.64, 3.92, 3.84 [15:36:15] !log reception@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php createLocalAccount.php --wiki metawiki Elli [15:36:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:57] [02WikiDiscover] 07Universal-Omega commented on pull request 03#36: Add (any) option to sort options dropdowns and default to it (T6577) - 13https://git.io/JqcAu [16:09:02] Hi, has SRE heard back from Outlook? I am unable to receive emails. [16:09:06] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqcAo [16:09:08] [02miraheze/WikiDiscover] 07Universal-Omega 03a5db74c - Remove unused function [16:09:09] [02WikiDiscover] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [16:09:10] [02WikiDiscover] 07Universal-Omega opened pull request 03#37: Remove unused function - 13https://git.io/JqcA6 [16:09:34] [02CreateWiki] 07Universal-Omega closed pull request 03#201: Re-add echo notifications for declined requests - 13https://git.io/JqZ1z [16:09:36] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqcAP [16:09:37] [02miraheze/CreateWiki] 07Universal-Omega 0347de36c - Re-add echo notifications for declined requests (#201) [16:09:39] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [16:09:40] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:10:06] miraheze/WikiDiscover - Universal-Omega the build passed. [16:10:35] miraheze/CreateWiki - Universal-Omega the build passed. [16:20:48] [02WikiDiscover] 07Universal-Omega closed pull request 03#37: Remove unused function - 13https://git.io/JqcA6 [16:20:49] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqcxs [16:20:51] [02miraheze/WikiDiscover] 07Universal-Omega 03fb6d1a2 - Remove unused function (#37) [16:20:52] [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:20:54] [02WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [16:21:19] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03revert-37-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqcxG [16:21:20] [02miraheze/WikiDiscover] 07Universal-Omega 03ab36427 - Revert "Remove unused function (#37)" [16:21:22] [02WikiDiscover] 07Universal-Omega created branch 03revert-37-Universal-Omega-patch-1 - 13https://git.io/vhUAp [16:21:23] [02WikiDiscover] 07Universal-Omega opened pull request 03#38: Revert "Remove unused function" - 13https://git.io/Jqcxn [16:21:27] [02WikiDiscover] 07Universal-Omega closed pull request 03#38: Revert "Remove unused function" - 13https://git.io/Jqcxn [16:21:29] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqcxc [16:21:30] [02miraheze/WikiDiscover] 07Universal-Omega 03b33bdcb - Revert "Remove unused function (#37)" (#38) [16:21:32] [02WikiDiscover] 07Universal-Omega deleted branch 03revert-37-Universal-Omega-patch-1 - 13https://git.io/vhUAp [16:21:33] [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03revert-37-Universal-Omega-patch-1 [16:21:50] Hi [16:21:56] miraheze/WikiDiscover - Universal-Omega the build passed. [16:22:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 191s [16:22:25] miraheze/WikiDiscover - Universal-Omega the build passed. [16:22:40] miraheze/WikiDiscover - Universal-Omega the build passed. [16:24:10] Hi RhinosF1! [16:24:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [16:25:29] Afternoon Universal_Omega [16:37:21] I'm getting persistent 502 bad gateways on `metawiki` [16:38:21] Unbreak Now! 502 Bad Gateway [16:38:45] Hi [16:38:49] Fixed? [16:38:51] PROBLEM - cp12 HTTPS on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.325 second response time [16:38:55] Looking [16:38:56] R4356th, nope [16:38:57] !sre [16:39:02] still getting 502s [16:39:05] PROBLEM - cp12 Varnish Backends on cp12 is WARNING: No backends detected. If this is an error, see readme.txt [16:39:15] Fine here, looks to be cp specific [16:39:15] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [16:39:19] Oof, ^ doesn't look good [16:39:21] Reception123: can you depool cp12 [16:39:23] Can't repro on Meta or my wiki [16:39:26] RhinosF1, yeah, could be specific to cp12 [16:39:27] dmehus: seems to be a localised cp12 issue, works for me [16:39:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [16:39:38] will depool and reboot [16:39:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [16:39:48] Reception123, SGTM and thanks :) [16:40:08] That's ca Reception123 for the code in dns [16:40:20] yup [16:40:28] Patch coming [16:40:39] what do you mean by ca in that comment, RhinosF1? [16:40:55] [02dns] 07RhinosF1 opened pull request 03#199: Depool ca/cp12 - 13https://git.io/Jqcpz [16:41:01] Reception123: ^ [16:41:05] dmehus: Canada [16:41:07] [02dns] 07Reception123 closed pull request 03#199: Depool ca/cp12 - 13https://git.io/Jqcpz [16:41:08] oh [16:41:09] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jqcpg [16:41:10] [02miraheze/dns] 07RhinosF1 036b34e72 - Update admin_state (#199) [16:41:17] It's the code that is used by dns [16:41:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is UNKNOWN: NRPE: Unable to read output [16:41:39] R E B O O [16:41:40] T [16:41:54] MarioMario456, it's coming [16:42:16] Reception123: that icinga doesn't sound good [16:42:37] Have you deployed? [16:42:41] !log rebooted cp12 [16:42:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:47] RhinosF1: yes, no errors were given [16:42:55] Reception123: okay [16:42:55] IT WORKS!!!!! [16:43:04] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [16:43:13] MarioMario456: we have depooled a cache proxy in Canada, it seems to have crashed [16:43:13] well, I guess that did it [16:43:15] PROBLEM - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is WARNING: WARNING - NGINX Error Rate is 57% [16:43:26] We might want to post an IR for that as cp12 was down for more than 5 minutes, possibly? [16:43:29] Reception123: let's give it 5. Check graylog for logs [16:43:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:43:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:43:39] dmehus: we're still mid icindent [16:43:49] RhinosF1, yeah definitely [16:43:51] yeah I'll wait a bit but it seems fine [16:43:53] just a comment for future [16:44:33] sometimes it's just a bad mood day for our servers [16:44:51] RECOVERY - cp12 HTTPS on cp12 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2214 bytes in 0.392 second response time [16:45:15] RECOVERY - cp12 HTTP 4xx/5xx ERROR Rate on cp12 is OK: OK - NGINX Error Rate is 6% [16:45:58] Reception123: try manually curling cp12 and make sure it don't crash again [16:46:10] But unless graylog says anything then I'm not sure [16:46:26] already curl'd, seems fine and I can't see anything weird in graylog [16:46:35] so I'll bring it back and hopefully it'll be fine [16:46:35] paladox, JohnLewis, SPF|Cloud: thoughts? [16:46:38] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqcpH [16:46:40] [02miraheze/dns] 07Reception123 037e0eb97 - Repool ca/cp12 [16:47:00] I do wonder why it suddenly decided to crash though [16:47:35] Grafana indicates a process crashed [16:47:41] I'd say [16:47:49] To see a sudden cpu/memory drop [16:49:23] https://www.irccloud.com/pastebin/XSuAQOmv/ [16:49:23] [ Snippet | IRCCloud ] - www.irccloud.com [16:49:27] I think I've found our culprit [16:49:48] That's it [16:49:51] OOM [16:49:54] the timing definitely matches, one minute before dmehus reported meta down [16:50:00] dmehus: cp12 ran out of memory [16:50:07] yeah, looks like it :( [16:50:15] heh and that's because I delayed reporting by 30 seconds or so [16:50:29] RhinosF1 and Reception123, ack [16:51:27] > sometimes it's just a bad mood day for our servers [16:51:27] Yeah, I just thought that potentially an IR would provide us with a centralized page with which to collate logs and provide other analysis as we look at potential solutions to prevent or mitigate reoccurence. IRs are a rather good thing in that way [16:52:20] Reception123: we should probably look at if we have enough memory or could purge it better [16:53:59] yeah, probably should hand that over to infra when they're around [16:55:07] Creating task [16:57:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.38, 3.78, 4.00 [16:57:59] https://phabricator.miraheze.org/T6952 Reception123 [16:58:00] [ ⚓ T6952 cp12 OOM'd March 12 2021 16:36 ] - phabricator.miraheze.org [16:58:18] ack [16:59:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.43, 4.02, 4.06 [16:59:31] Main points would be: why didn't it alert/depool automatically before user visible? Could we limit memory usage better / clear old stuff from memory? / do we need more resources? [17:02:48] yeah [17:03:20] and I noticed paladox and John weren't auto-added to that Infrastructure (SRE) task. They may want to review their Herald rules, following the project split? [17:07:20] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.58, 2.76, 1.53 [17:09:20] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.96, 2.06, 1.43 [17:41:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.31, 3.71, 3.99 [17:43:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.13, 4.00, 4.06 [17:51:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.72, 3.77, 3.96 [17:53:05] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.12, 3.94, 4.00 [18:51:11] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCTg [18:51:13] [02miraheze/CreateWiki] 07Universal-Omega 034dac6b3 - fix CreateWikiPurposes [18:52:40] [02CreateWiki] 07Universal-Omega opened pull request 03#203: fix CreateWikiPurposes - 13https://git.io/JqCTr [18:53:16] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCTX [18:53:18] [02miraheze/CreateWiki] 07Universal-Omega 03b023f41 - fix CreateWikiPurposes (#203) [18:53:21] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:53:46] miraheze/CreateWiki - Universal-Omega the build passed. [18:53:48] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [18:54:30] miraheze/CreateWiki - Universal-Omega the build passed. [18:55:39] [02CreateWiki] 07Universal-Omega closed pull request 03#203: fix CreateWikiPurposes - 13https://git.io/JqCTr [18:55:44] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [20:24:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCqH [20:24:13] [02miraheze/mw-config] 07Universal-Omega 03debc32c - Move configs from LocalWiki.php to LocalSettings.php [20:24:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [20:24:32] [02mw-config] 07Universal-Omega opened pull request 03#3774: Move configs from LocalWiki.php to LocalSettings.php - 13https://git.io/JqCq5 [20:24:45] [02mw-config] 07Universal-Omega edited pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [20:25:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCqA [20:25:31] [02miraheze/mw-config] 07Universal-Omega 030fa4b3b - Update LocalExtensions.php [20:25:32] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [20:25:40] miraheze/mw-config - Universal-Omega the build has errored. [20:26:36] miraheze/mw-config - Universal-Omega the build has errored. [20:26:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCmf [20:26:48] [02miraheze/mw-config] 07Universal-Omega 03d869908 - Update LocalSettings.php [20:26:49] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [20:26:51] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [20:27:51] miraheze/mw-config - Universal-Omega the build has errored. [20:27:56] miraheze/mw-config - Universal-Omega the build has errored. [20:28:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCmI [20:28:20] [02miraheze/mw-config] 07Universal-Omega 03a70f888 - Update LocalSettings.php [20:28:22] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [20:29:21] miraheze/mw-config - Universal-Omega the build passed. [20:40:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCm9 [20:41:00] [02miraheze/mediawiki] 07Universal-Omega 03df95741 - Update MirahezeMagic [20:41:01] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [20:41:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.52, 3.69, 3.96 [20:42:44] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JqCm7 [20:42:45] [02miraheze/mediawiki] 07Universal-Omega 03d67c53a - Update CreateWiki [20:42:47] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [20:43:45] [02mediawiki] 07Universal-Omega opened pull request 03#1276: Update MirahezeMagic - 13https://git.io/JqCmb [20:44:30] [02mediawiki] 07Universal-Omega opened pull request 03#1277: Update CreateWiki - 13https://git.io/JqCmx [20:44:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JqCmj [20:44:50] [02miraheze/mediawiki] 07Universal-Omega 035d6ef39 - Update WikiDiscover [20:44:52] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbL5b [20:45:29] [02mediawiki] 07Universal-Omega opened pull request 03#1278: Update WikiDiscover - 13https://git.io/JqCYU [20:46:20] [02mediawiki] 07Universal-Omega closed pull request 03#1278: Update WikiDiscover - 13https://git.io/JqCYU [20:46:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqCYL [20:46:23] [02miraheze/mediawiki] 07Universal-Omega 03911d14a - Update WikiDiscover (#1278) [20:46:24] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbL5b [20:46:26] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [20:47:07] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.68, 3.94, 3.95 [20:50:47] [02mediawiki] 07Universal-Omega closed pull request 03#1277: Update CreateWiki - 13https://git.io/JqCmx [20:50:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqCYE [20:50:50] [02miraheze/mediawiki] 07Universal-Omega 03417c800 - Update CreateWiki (#1277) [20:50:52] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [20:50:53] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [21:03:20] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.54, 3.35, 1.74 [21:04:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqCOR [21:04:37] [02miraheze/mw-config] 07Universal-Omega 03adc6f88 - Re-enable wgCreateWikiPurposes [21:05:16] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4233 MB (10% inode=96%); [21:05:17] [02mediawiki] 07Universal-Omega closed pull request 03#1276: Update MirahezeMagic - 13https://git.io/JqCmb [21:05:19] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JqCOa [21:05:20] [02miraheze/mediawiki] 07Universal-Omega 0395bdec8 - Update MirahezeMagic (#1276) [21:05:22] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:05:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:05:24] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.42, 2.48, 1.62 [21:05:41] miraheze/mw-config - Universal-Omega the build passed. [21:06:22] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:11:02] [02mw-config] 07Universal-Omega closed pull request 03#3742: Move SiteNoticeAfter hooks from mw-config to MirahezeMagic - 13https://git.io/JtbGG [21:11:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqC3J [21:11:05] [02miraheze/mw-config] 07Universal-Omega 038a0ddcd - Move SiteNoticeAfter hooks from mw-config to MirahezeMagic (#3742) [21:11:07] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://git.io/vbvb3 [21:11:08] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [21:12:06] miraheze/mw-config - Universal-Omega the build passed. [21:14:58] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*, jbr*, and test3 [21:15:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:08] https://meta.miraheze.org/w/index.php?title=User_talk%3ASouthparkfan&type=revision&diff=166793&oldid=166780 [21:18:09] [ Difference between revisions of "User talk:Southparkfan" - Miraheze Meta ] - meta.miraheze.org [21:21:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 3.48, 3.59, 3.95 [21:27:04] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 4.16, 3.81, 3.94 [21:33:11] SPF|Cloud, ty [21:34:18] SPF|Cloud https://phabricator.miraheze.org/T6952#137381 [21:34:20] [ ⚓ T6952 cp12 OOM'd March 12 2021 16:36 ] - phabricator.miraheze.org [21:35:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.88, 7.48, 5.20 [21:35:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.45, 7.61, 5.12 [21:36:22] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 1 minute ago with 0 failures [21:40:03] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [21:40:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCs2 [21:40:06] [02miraheze/mw-config] 07Universal-Omega 037fd7b8f - Update LocalSettings.php [21:40:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.43, 2.61 [21:41:05] miraheze/mw-config - Universal-Omega the build passed. [21:42:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCsP [21:42:13] [02miraheze/mw-config] 07Universal-Omega 0363ca64f - Update LocalSettings.php [21:42:15] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [21:43:02] PROBLEM - services3 APT on services3 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (1 critical updates). [21:43:16] miraheze/mw-config - Universal-Omega the build passed. [21:43:35] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:44:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.59, 2.87 [21:44:51] paladox: what does https://phabricator.miraheze.org/T6952#137381 mean? [21:44:52] [ ⚓ T6952 cp12 OOM'd March 12 2021 16:36 ] - phabricator.miraheze.org [21:45:02] No space left on device [21:45:05] How do we prevent / mitigate in future [21:45:09] Disk? [21:45:34] i only found the log :) [21:45:42] PROBLEM - cloud4 APT on cloud4 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [21:45:48] Ack ok [21:45:58] i guess related to https://github.com/miraheze/puppet/blob/master/modules/varnish/manifests/init.pp#L29 [21:45:58] [ puppet/init.pp at master · miraheze/puppet · GitHub ] - github.com [21:47:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JqCGf [21:47:14] [02miraheze/mw-config] 07Universal-Omega 0342d7701 - Update LocalSettings.php [21:47:15] [02mw-config] 07Universal-Omega synchronize pull request 03#3774: Move configs from LocalExtensions.php to LocalSettings.php - 13https://git.io/JqCq5 [21:47:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.82, 7.92, 7.00 [21:48:01] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.79, 2.75, 2.71 [21:48:15] miraheze/mw-config - Universal-Omega the build passed. [21:49:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.50, 7.78, 7.20 [21:49:50] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 22 packages available for upgrade (1 critical updates). [21:50:12] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [21:51:36] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.70, 6.27, 6.57 [21:51:48] PROBLEM - cloud3 APT on cloud3 is CRITICAL: APT CRITICAL: 97 packages available for upgrade (1 critical updates). [21:51:57] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (1 critical updates). [21:51:59] PROBLEM - db11 APT on db11 is CRITICAL: APT CRITICAL: 62 packages available for upgrade (1 critical updates). [21:52:11]