[00:05:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.97, 3.67, 2.93 [00:07:38] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.92, 3.25, 2.87 [00:45:54] !log depool mw[456] [00:45:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:56:22] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.60, 5.43, 2.87 [00:58:22] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.31, 5.95, 3.40 [01:05:23] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.24, 7.12, 4.75 [01:09:22] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.55, 7.80, 5.58 [01:13:18] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.16, 6.74, 5.62 [01:17:14] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.83, 6.87, 5.92 [01:19:13] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.60, 6.63, 5.94 [01:27:10] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.34, 7.30, 6.37 [01:31:11] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.57, 7.73, 6.79 [01:33:09] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.64, 7.86, 6.93 [01:37:09] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.56, 7.91, 7.18 [01:45:10] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 11.74, 8.24, 7.38 [01:47:09] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.63, 7.77, 7.30 [01:49:07] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.36, 8.39, 7.58 [02:03:12] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.02, 7.10, 7.77 [02:11:07] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.17, 5.07, 6.57 [06:25:34] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3150 MB (13% inode=94%); [12:09:32] !log install golang on glusterfs1 to build the gluster prometheus exporter [12:09:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:19:59] PROBLEM - glusterfs1 SSH on glusterfs1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:05] PROBLEM - glusterfs1 Current Load on glusterfs1 is CRITICAL: connect to address 81.4.100.90 port 5666: Connection refusedconnect to host 81.4.100.90 port 5666: Connection refused [12:21:46] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: connect to address 81.4.100.90 port 5666: Connection refusedconnect to host 81.4.100.90 port 5666: Connection refused [12:21:49] PROBLEM - glusterfs1 Disk Space on glusterfs1 is CRITICAL: connect to address 81.4.100.90 port 5666: Connection refusedconnect to host 81.4.100.90 port 5666: Connection refused [12:21:57] RECOVERY - glusterfs1 SSH on glusterfs1 is OK: SSH OK - OpenSSH_7.9p1 Debian-10 (protocol 2.0) [12:23:47] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:23:49] RECOVERY - glusterfs1 Disk Space on glusterfs1 is OK: DISK OK - free space: / 261709 MB (83% inode=95%); [12:26:00] RECOVERY - glusterfs1 Current Load on glusterfs1 is OK: OK - load average: 0.58, 3.14, 3.12 [12:26:11] that was me [12:30:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.77, 1.60, 1.38 [12:32:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.33, 1.53, 1.37 [12:58:55] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.77, 1.63, 1.47 [13:02:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.52, 1.69, 1.53 [13:06:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.87, 1.83, 1.62 [13:08:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.20, 1.98, 1.70 [13:10:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.94, 1.96, 1.73 [13:14:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.09, 1.58, 1.63 [14:22:25] .at 21:00 set private Chan invites [14:22:25] RhinosF1: Okay, I will set the reminder for: 2019-09-13 - 19:59:59UTC [14:27:13] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.89, 6.89, 5.56 [14:33:10] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.43, 6.79, 6.03 [16:52:55] Will post on Community Notice board for feedback but does anybody mind me adding an option to Meta's MediaWiki: Emailpagetext to allow adding custom notes below the standard message. Or does anyone have better suggestions for wording? [17:02:55] Reception123, Voidwalker: thoughts? [17:09:33] don't mind [17:17:32] Reception123: to clarify, if you don't have an email notice then it WILL NOT have any impact nor will the original message be affected [17:18:11] ok [17:29:22] https://meta.miraheze.org/w/index.php?title=Community_noticeboard&diff=82643&oldid=82534 [17:29:23] [ Difference between revisions of "Community noticeboard" - Miraheze Meta ] - meta.miraheze.org [17:59:32] paladox: 503 backend error! [17:59:39] oh [17:59:47] not db4 [17:59:53] had to rule that one out first [18:00:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYBO [18:00:14] [02miraheze/services] 07MirahezeSSLBot 031e7aa66 - BOT: Updating services config for wikis [18:00:36] it's not lizardfs [18:00:45] unless it just had a quick blip and i missed it [18:00:57] Reception123 things are back. [18:15:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYBz [18:15:09] [02miraheze/services] 07MirahezeSSLBot 03b55d3c1 - BOT: Updating services config for wikis [18:15:34] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2636 MB (10% inode=94%); [18:20:48] Hi! Here is the list of currently open high priority tasks on Phabricator [18:20:55] No updates for 13 days - https://phabricator.miraheze.org/T4504 - Renew *.miraheze.org cert - authored by Reception123, assigned to Southparkfan [18:21:10] SPF|Cloud: ^ [18:31:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:33:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:27:43] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.06, 1.81, 1.45 [19:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYRO [19:30:12] [02miraheze/services] 07MirahezeSSLBot 03142466d - BOT: Updating services config for wikis [19:31:31] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.83, 1.98, 1.60 [19:41:01] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.42, 1.64, 1.58 [20:00:01] RhinosF1: set private Chan invites [20:01:41] Already done [20:17:57] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 6.31, 4.86 [20:23:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.24, 7.75, 6.06 [20:25:55] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.71, 6.43, 5.79 [20:40:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYRx [20:40:12] [02miraheze/services] 07MirahezeSSLBot 034653abd - BOT: Updating services config for wikis [20:43:28] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:43:31] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:43:35] Hello VisbyStar! If you have any questions feel free to ask and someone should answer soon. [20:43:43] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [20:43:46] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:43:59] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [20:44:02] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:44:04] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 215 failures. Last run 2 minutes ago with 215 failures. Failed resources (up to 3 shown) [20:44:06] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 2 minutes ago with 32 failures. Failed resources (up to 3 shown) [20:44:10] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:44:11] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 210 failures. Last run 2 minutes ago with 210 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [20:44:18] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 210 failures. Last run 2 minutes ago with 210 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [20:44:19] Go away puppet [20:44:26] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:44:37] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 3 minutes ago with 16 failures. Failed resources (up to 3 shown) [20:44:38] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 138 failures. Last run 2 minutes ago with 138 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades] [20:44:39] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:44:47] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:45:04] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [20:45:05] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:45:07] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 198 failures. Last run 3 minutes ago with 198 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades] [20:45:21] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:45:54] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 210 failures. Last run 3 minutes ago with 210 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [20:53:31] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:53:43] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:53:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:53:47] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:53:59] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:54:05] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:54:10] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:54:37] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:54:39] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:54:48] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:04] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:05] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:06] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:55:21] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:28] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:49] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:55:58] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:56:02] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:56:13] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:56:16] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:56:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:56:44] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:59:19] !log depool mw1 [21:59:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:02:26] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures [22:02:34] that's me [22:02:40] k [22:02:51] i needed to disable it quickly hence why i didn't put paladox [22:03:03] np [22:04:23] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.93, 6.10, 3.93 [22:06:22] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.30, 6.23, 4.23 [22:06:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.91, 7.22, 5.58 [22:06:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 37 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/var/lib/glusterd/secure-access] [22:08:22] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.99, 6.23, 4.48 [22:08:26] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.03, 7.89, 6.03 [22:10:28] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:26:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.26, 7.13, 7.88 [22:34:20] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 12.11, 7.92, 7.61 [22:35:28] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/var/lib/glusterd/secure-access] [22:37:25] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:40:17] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.14, 7.62, 7.66 [22:46:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.79, 5.77, 6.75 [22:53:42] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [22:53:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:55:36] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:55:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online