[00:04:39] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [00:12:40] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:04:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [01:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNpQ [01:05:09] [02miraheze/services] 07MirahezeSSLBot 03e247190 - BOT: Updating services config for wikis [01:22:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:54:45] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [02:02:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:04:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],Service[ssh],Package[exim4] [03:12:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:54:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-8140] [04:02:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:34:14] PROBLEM - wiki.candela.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.candela.digital' expires in 15 day(s) (Wed 15 Jan 2020 04:31:33 AM GMT +0000). [04:34:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAv4 [04:34:28] [02miraheze/ssl] 07MirahezeSSLBot 036ae8d8a - Bot: Update SSL cert for wiki.candela.digital [04:42:17] RECOVERY - wiki.candela.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candela.digital' will expire on Sun 29 Mar 2020 03:34:21 AM GMT +0000. [04:44:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [04:49:35] PROBLEM - wiki.itsuali.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.itsuali.tk' expires in 15 day(s) (Wed 15 Jan 2020 04:46:01 AM GMT +0000). [04:49:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAva [04:49:50] [02miraheze/ssl] 07MirahezeSSLBot 03d62f64d - Bot: Update SSL cert for wiki.itsuali.tk [05:02:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [05:03:35] RECOVERY - wiki.itsuali.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.itsuali.tk' will expire on Sun 29 Mar 2020 03:49:43 AM GMT +0000. [05:24:39] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [05:52:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:55:34] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:03:53] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:04:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [06:12:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:26:02] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3362 MB (13% inode=94%); [06:34:40] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],Service[ssh],Package[exim4] [06:42:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:04:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [07:22:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:04:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [08:12:38] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [08:34:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [09:22:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:34:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Service[puppetserver],Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [09:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAkX [09:40:09] [02miraheze/services] 07MirahezeSSLBot 033aba51e - BOT: Updating services config for wikis [09:52:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:04:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [10:12:41] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [11:04:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [11:42:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [12:14:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[salt-minion] [12:22:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [12:34:39] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-8081] [12:42:41] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:01:10] [02puppet] 07RhinosF1 opened pull request 03#1172: Update donate@ - 13https://git.io/JeAmk [13:01:58] Leave to SPF|Cloud pls [13:04:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [13:12:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client],Package[openssh-server] [13:32:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:07:15] paladox. JohnLewis: around? If so, ping me in #miraheze-meetings [14:07:26] paladox: ^ [14:15:27] PuppyKun: ? [14:24:30] Hi hispano76 [14:24:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [14:32:39] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:33:15] Hi Jianhui67 [14:33:28] hello [14:35:46] Jianhui67: just to inform you, I started an impromptu #miraheze-meetings about a face to face event for users [14:36:07] alright thanks [14:37:58] hi RhinosF1 :) [14:54:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [15:02:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:14:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [15:32:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:41:23] RhinosF1: https://meta.miraheze.org/wiki/Talk:Miraheze_Meetings/Calendar/2020/Jan/Miramania [15:41:24] [ Talk:Miraheze Meetings/Calendar/2020/Jan/Miramania - Miraheze Meta ] - meta.miraheze.org [15:42:22] quirc: thx [15:42:44] We could all video in for a lot or audio call [15:43:08] But if there’s groups of users we could set up for everyone to gathet in certain areas [15:43:22] Maybe have a UK room calling into a US one [15:47:15] If we use Discord, you can auto-mute everyone but the speaker. Then as people want to make comments, they can post in a text channel or the admin can unmute them for a moment. [15:49:09] Yeah [15:49:40] It’d be nice to meet everyone [15:54:38] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [16:22:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:24:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [17:32:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:44:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client] [17:51:09] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeACR [17:51:11] [02miraheze/puppet] 07Reception123 03a82875a - add robla to board [17:51:40] ^ robla [17:56:02] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [17:58:02] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2651 MB (11% inode=94%); [18:02:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:02] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%); [18:24:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [18:32:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:54:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter] [19:12:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:21:39] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21976 MB (5% inode=95%); [19:24:17] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client] [19:24:26] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:32:28] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:45:23] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [19:46:04] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:18] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:56] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [20:42:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:00] [02puppet] 07labster commented on pull request 03#1172: Update donate@ - 13https://git.io/JeA4y [20:54:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Exec[ufw-default-deny],Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server] [21:24:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:24:55] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:33] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [21:26:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:26:55] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1366 bytes in 0.035 second response time [21:26:55] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1366 bytes in 0.047 second response time [21:27:51] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:28:04] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:28:30] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 47% [21:29:02] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18617 bytes in 0.348 second response time [21:29:15] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 0.294 second response time [21:29:51] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [21:30:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [21:30:26] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 26% [21:30:38] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:31:00] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18618 bytes in 0.783 second response time [21:33:59] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 415 failures. Last run 2 minutes ago with 415 failures. Failed resources (up to 3 shown) [21:34:18] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [21:34:26] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown) [21:34:33] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 47 failures. Last run 2 minutes ago with 47 failures. Failed resources (up to 3 shown) [21:34:39] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [21:34:45] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [21:34:52] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown) [21:34:54] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 422 failures. Last run 3 minutes ago with 422 failures. Failed resources (up to 3 shown) [21:35:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [21:35:08] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [21:35:20] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:35:33] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 425 failures. Last run 3 minutes ago with 425 failures. Failed resources (up to 3 shown) [21:35:42] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 424 failures. Last run 3 minutes ago with 424 failures. Failed resources (up to 3 shown) [21:35:58] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:36:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:36:36] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 141 failures. Last run 3 minutes ago with 141 failures. Failed resources (up to 3 shown): File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common] [21:36:49] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 200 failures. Last run 3 minutes ago with 200 failures. Failed resources (up to 3 shown): File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common] [21:42:04] [02puppet] 07RhinosF1 synchronize pull request 03#1172: Update donate@ - 13https://git.io/JeAmk [21:42:38] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [21:42:42] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [21:42:50] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [21:42:52] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:43:01] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:43:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [21:43:33] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [21:43:50] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:44:09] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:44:19] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [21:44:21] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:44:36] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [21:44:37] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:44:38] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:44:45] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [21:44:54] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:45:48] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:48:02] [02puppet] 07paladox reviewed pull request 03#1172 commit - 13https://git.io/JeARY [21:50:57] [02puppet] 07RhinosF1 reviewed pull request 03#1172 commit - 13https://git.io/JeARc [21:51:48] paladox: github’s diff being weird [21:52:21] it's not being wierd :) [21:52:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [21:52:59] paladox: Reception123 added it [21:53:05] yup [21:53:13] the rebase didn't work as planned it seems? [21:53:31] paladox: it’ll be fine [21:53:35] no [21:53:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [21:53:55] i'm not merging it till that is removed :) [21:54:25] paladox: i said leave for SPF|Cloud before merging anyway [21:54:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:55:15] [02puppet] 07RhinosF1 commented on pull request 03#1172: Update donate@ - 13https://git.io/JeAR4 [21:55:24] anyways i think that alias can be removed actually (per the update to the finance page removing donate@) [21:55:32] [02puppet] 07RhinosF1 commented on pull request 03#1172: Update donate@ - 13https://git.io/JeARB [21:55:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:55:53] paladox: was it? [21:56:20] SPF|Cloud can restore it if he wants so let him review [21:56:38] RhinosF1 https://meta.miraheze.org/w/index.php?title=Donate&type=revision&diff=92961&oldid=92956 [21:56:41] [ Difference between revisions of "Donate" - Miraheze Meta ] - meta.miraheze.org [21:57:18] paladox: https://github.com/miraheze/puppet/pull/1172#issuecomment-569791948 - it could come back [21:57:20] [ Update donate@ by RhinosF1 · Pull Request #1172 · miraheze/puppet · GitHub ] - github.com [21:58:16] ok... which we can always re-add it back when we do. [21:58:23] JohnLewis ^ [22:01:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:02:05] paladox: that’ll be Miraheze Limited to sort as they have the account. SPF|Cloud has been told. Why not wait a day or two, it does no harm. He might also have other news in a PM. [22:03:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:03:56] ok [22:08:32] [02puppet] 07RhinosF1 synchronize pull request 03#1172: Update donate@ - 13https://git.io/JeAmk [22:34:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [22:36:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:38:32] .status mhmeta Off [22:38:36] RhinosF1: Updating User:RhinosF1/Status to Off! [22:38:43] RhinosF1: Updated! [22:42:59] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:43:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:43:27] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:44:57] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:45:00] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15198 bytes in 1.728 second response time [22:45:23] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18618 bytes in 1.304 second response time [22:45:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:46:37] !log restart nginx and php7.3-fpm on mw* and lizardfs6 [22:47:06] !log restart nginx on cp* [22:47:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:47:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:54:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [23:02:50] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:04:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:04:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:10:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:12:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:12:51] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:23] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:56] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15190 bytes in 6.056 second response time [23:16:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:16:25] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18619 bytes in 4.699 second response time [23:16:51] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:32:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [23:34:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online