[00:05:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [00:08:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [00:11:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:12:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [01:43:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:03:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:03:58] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [02:04:04] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown) [02:04:27] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown) [02:04:43] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 392 failures. Last run 2 minutes ago with 392 failures. Failed resources (up to 3 shown) [02:04:43] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 403 failures. Last run 2 minutes ago with 403 failures. Failed resources (up to 3 shown) [02:04:56] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [02:04:57] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [02:05:16] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 398 failures. Last run 2 minutes ago with 398 failures. Failed resources (up to 3 shown) [02:05:19] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [02:05:33] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:05:48] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [02:05:50] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 398 failures. Last run 3 minutes ago with 398 failures. Failed resources (up to 3 shown) [02:06:03] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 402 failures. Last run 3 minutes ago with 402 failures. Failed resources (up to 3 shown) [02:06:49] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 39 failures. Last run 4 minutes ago with 39 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local],File[/usr/local/bin/prometheus-gdnsd-stats],File[/usr/lib/nagios/plugins/check_gdnsd_datacenters],File[wildcard.miraheze.org] [02:06:54] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 252 failures. Last run 3 minutes ago with 252 failures. Failed resources (up to 3 shown): File[/home/nagiosre],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params],File[/usr/local/bin/nginx-prometheus-exporter] [02:07:08] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 169 failures. Last run 3 minutes ago with 169 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/salt-common],File[/etc/logrotate.d/puppet],File[/etc/sudoers],File[/home/nagiosre] [02:13:10] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:13:20] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:13:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [02:13:51] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [02:13:57] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [02:14:00] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [02:14:03] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:31] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:48] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:14:50] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:55] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:15:06] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [02:15:10] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:15:22] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:15:22] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:15:52] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:16:27] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:22:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:23:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:24:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:25:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:33:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [02:44:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [02:46:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:51:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:36:30] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1424 MB (5% inode=94%); [03:53:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [04:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:33:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [04:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [05:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [05:31:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:13:57] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:21:57] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-udp-from-any-to-any-port-53] [06:26:29] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2396 MB (9% inode=94%); [06:31:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:43:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:51:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [08:03:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:11:55] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [08:30:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jey3s [08:30:11] [02miraheze/services] 07MirahezeSSLBot 038056006 - BOT: Updating services config for wikis [09:44:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [09:52:39] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:14:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [10:21:55] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [10:33:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [10:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [11:32:02] PROBLEM - db5 Current Load on db5 is CRITICAL: CRITICAL - load average: 4.55, 3.38, 1.99 [12:22:02] PROBLEM - db5 Current Load on db5 is WARNING: WARNING - load average: 0.51, 1.22, 3.83 [12:26:02] RECOVERY - db5 Current Load on db5 is OK: OK - load average: 0.47, 0.88, 3.08 [12:33:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 2 minutes ago with 15 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [12:41:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:33:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [13:53:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:29:52] [02puppet] 07Pix1234 opened pull request 03#1159: Add owen to the donate alias - 13https://git.io/Jey8Z [14:36:43] [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_33 [+0/-0/±4] 13https://git.io/Jey8R [14:36:44] [02miraheze/mediawiki] 07Pix1234 0377c3b3a - Update various ext from upstream [14:38:12] !log force puppet runs on mw[123] to deploy https://git.io/Jey8R [14:38:13] [ Comparing 072566c4d2cb...77c3b3a4d2cc · miraheze/mediawiki · GitHub ] - git.io [14:38:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:43:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [14:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jey86 [14:45:09] [02miraheze/services] 07MirahezeSSLBot 032435fc6 - BOT: Updating services config for wikis [14:45:33] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:19] !log rebuild lc on mw[13]; MW2 puppet run failed, paladox is fixing, will run rebuild lc after [14:47:29] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:47:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:49:54] !log rebuild LC on mw2 [14:50:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:51:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:56:09] PROBLEM - bacula1 Bacula Databases db5 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:24] [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_34 [+0/-0/±4] 13https://git.io/Jey8A [14:56:25] [02miraheze/mediawiki] 07Pix1234 03a04b0cb - Update various ext from upstream [14:57:29] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:01] !log rebooted db5 after networking stoped after running "sudo service networking restart" [14:58:15] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:58:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [14:58:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:58:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [14:58:47] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 5 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx],Service[nginx-prometheus-exporter] [14:58:49] PROBLEM - bacula1 Bacula Databases db5 on bacula1 is WARNING: WARNING: Full, 2161 files, 71.19GB, 2019-11-20 06:10:00 (2.3 weeks ago) [14:59:25] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.684 second response time [15:00:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:00:19] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 4.966 second response time [15:00:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:01:59] !log rebuild LC on test1 [15:02:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:04:57] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:07:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/Jey4I [15:07:31] [02miraheze/puppet] 07paladox 03c9a7817 - Delete vpncloud_1.0.0_amd64.deb [15:08:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jey4t [15:08:05] [02miraheze/puppet] 07paladox 0392447c5 - Update site.pp [15:24:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [15:26:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:13:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-3506] [16:57:12] .at 21:00 update phone [16:57:13] RhinosF1: Okay, I will set the reminder for: 2019-12-06 - 20:59:59UTC [17:11:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:34:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [17:41:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:32:50] Zppix, paladox: can someone get cloud services to check https://phabricator.wikimedia.org/T240036 [18:32:51] [ ⚓ T240036 Unable to connect to MySQL (enwiki.labsdb) ] - phabricator.wikimedia.org [18:33:30] i doin't deal with toolslabs [18:33:41] not atm im busy [18:34:26] paladox: if you can get into #wikimedia-cloud just use !_help and mention the task [18:34:34] oh, no [18:34:37] you need to do that [18:34:44] i rather not spam the team [18:34:55] paladox: remember, I can’t [18:34:59] ok [18:35:04] well i'm not. [18:35:10] Since then i could be banned [18:35:41] lol your not gonna get banned for using the command for its intended purpose [18:35:55] yes i would for helping RhinosF1 avoid his ban [18:36:11] since that task has nothing to do with me, i carn't say i'm asking on my behalf. [18:36:35] My concern is that it’s more sinister than it looks and the labsdb has failed [18:36:49] Ok, then wait for someone to notice? [18:36:57] Why the need to rush if you carn't ask your self? [18:38:24] Ok [19:03:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:31:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:03:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [20:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [20:33:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [20:51:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:59:59] RhinosF1: update phone [21:15:30] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:17:26] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.005 second response time [21:53:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [22:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:24:03] paladox: see discord [23:05:41] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-redis],Service[rsyslog] [23:13:53] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:22:34] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:24:36] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 3.114 second response time [23:25:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:31:43] Hello Examknow9! If you have any questions, feel free to ask and someone should answer soon.