[00:06:41] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: Full, 4315 files, 8.996MB, 2019-11-24 00:05:00 (2.3 weeks ago) [00:24:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [00:32:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [00:44:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Exec[git_pull_dns] [01:01:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [01:23:42] ping paladox [01:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server],Package[openssh-client] [01:24:36] Hi? [01:25:32] I wanted you to perform this command php updateArticleCount.php --update in UcroniasWiki to see if Special:Stadistics is updated or is still counting badly. https://www.mediawiki.org/wiki/Manual:UpdateArticleCount.php/es If you don't have time, no problem.paladox [01:25:33] [ Manual:UpdateArticleCount.php - MediaWiki ] - www.mediawiki.org [01:25:53] https://ucronias.miraheze.org/wiki/Especial:Estad%C3%ADsticas [01:25:54] [ Estadísticas - Ucronías ] - ucronias.miraheze.org [01:25:57] I’m not currently at my mac [01:26:02] I’m on my mobile [01:27:01] ah, ok :) [01:27:19] Would you be able to file a task please? :) [01:28:55] ok :) [01:31:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:37:12] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[allthetropes.org],File[allthetropes.org_private] [01:40:41] done https://phabricator.miraheze.org/T4974 [01:40:42] [ ⚓ T4974 Incorrect statistics in UcroniasWiki ] - phabricator.miraheze.org [01:45:09] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:33:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Exec[ufw-allow-udp-from-any-to-any-port-53],Exec[ufw-allow-tcp-from-any-to-any-port-53] [02:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:54:08] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1444 MB (5% inode=94%); [02:56:08] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1465 MB (6% inode=94%); [03:18:08] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1438 MB (5% inode=94%); [03:33:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [03:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [04:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:43:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [05:53:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:26:08] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2407 MB (9% inode=94%); [06:34:15] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:42:12] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [07:01:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:30:52] You've only got another few hours to nominate yourself for the election commission. See https://meta.miraheze.org/wiki/Community_Directors/Election_Commission/2019_Election - It's already overdue but I'm rushing out for a bit :slight_smile: [07:30:53] [ Community Directors/Election Commission/2019 Election - Miraheze Meta ] - meta.miraheze.org [07:33:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:40:56] Hello xahon! If you have any questions, feel free to ask and someone should answer soon. [07:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:42:34] Hello. When new wiki request will be handled? [08:34:07] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:35:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [08:35:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [08:37:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:37:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [08:51:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [09:02:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [09:04:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [09:07:49] PROBLEM - wiki.8b8t.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:32] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:10:34] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:11:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [09:11:39] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.701 second response time [09:12:29] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [09:12:42] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 8.694 second response time [10:10:16] RECOVERY - wiki.8b8t.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.8b8t.com' will expire on Fri 21 Feb 2020 05:25:26 AM GMT +0000. [11:13:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [11:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [12:13:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [12:40:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je9OY [12:40:09] [02miraheze/services] 07MirahezeSSLBot 03e3b1fc2 - BOT: Updating services config for wikis [12:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [13:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:01:10] PROBLEM - glp.ink - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [14:04:42] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Thu 26 Dec 2019 02:00:52 PM GMT +0000). [14:04:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je9su [14:04:57] [02miraheze/ssl] 07MirahezeSSLBot 037e9e2d4 - Bot: Update SSL cert for wiki.mxlinuxusers.de [14:12:39] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Mon 09 Mar 2020 01:04:49 PM GMT +0000. [14:46:08] anyone around to fix some broken renames again? [14:46:14] got two this time: https://meta.miraheze.org/wiki/Special:GlobalRenameProgress [14:46:15] [ Global rename progress - Miraheze Meta ] - meta.miraheze.org [15:03:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [15:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [15:53:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [15:55:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:05:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [16:11:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:30:58] Vote for the first ever Miraheze Community Directors Election Commission via https://meta.miraheze.org/wiki/Community_Directors/Election_Commission/2019_Election/Vote#Voting. Everyone is encouraged to vote and help decide who gets to ensure you get a chance to pick the community seats for the Miraheze Board. Any questions, please contact us. [16:31:02] [ Community Directors/Election Commission/2019 Election/Vote - Miraheze Meta ] - meta.miraheze.org [16:48:35] * hispano76 gretting [16:49:12] hispano76: hi [16:54:23] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [16:54:55] greeting* :| [16:55:28] hispano76: thanks for the fix :) [16:55:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:55:56] no problem RhinosF1 :) [16:56:47] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 2.832 second response time [16:58:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:59:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:02:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:03:37] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:04:06] Approves translation Sitenotice RhinosF1 [17:05:32] hispano76: i have no clue how to [17:05:46] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 9.802 second response time [17:10:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:14:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [17:14:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [17:16:01] hispano76: I think I published them [17:16:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:23:05] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:13] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 8.820 second response time [17:25:49] Ok RhinosF1 thanks [17:26:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:32:46] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:33:04] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:33:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:33:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [17:33:28] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:34:45] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.131 second response time [17:35:07] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 3.989 second response time [17:35:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:35:21] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:45:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [17:47:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:55:23] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [17:57:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [17:58:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [18:03:29] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:05:36] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:14:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [18:15:19] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:17:23] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 4.899 second response time [18:19:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:23:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:25:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:28:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:35:31] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [18:37:37] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.091 second response time [18:38:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:43:01] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:24] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:44:37] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:45:06] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.947 second response time [18:46:26] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 4.155 second response time [18:46:39] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 4.538 second response time [18:47:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:48:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:56:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [18:59:07] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=metawiki /srv/mediawiki/w/extensions/OATHAuth/sql/mysql/patch-add_generic_fields.sql (mw1) [18:59:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:00:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:00:13] !log turns out that, that file does not exist on mw1 but rather test1, so ran it on test1 [19:01:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:04:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:04:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:05:42] !log MariaDB [mhglobal]> ALTER TABLE /*_*/oathauth_users [19:05:43] -> ADD module VARCHAR( 255 ) NOT NULL, [19:05:43] -> ADD data BLOB NULL; [19:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:06:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:07:02] !log sudo -u www-data php /srv/mediawiki/w/extensions/OATH*/maint*/updateTOTPToMultipleKeys.php --wiki=metawiki (test1) [19:07:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:08:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:13:10] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [19:13:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je98y [19:13:54] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [19:13:55] [02miraheze/mw-config] 07paladox 03298a65d - Remove EducationProgram extension Extension now unmaintained and does not support 1.34+ [19:13:56] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:13:58] [02mw-config] 07paladox opened pull request 03#2823: Remove EducationProgram extension - 13https://git.io/Je989 [19:14:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je98H [19:14:27] [02miraheze/mw-config] 07paladox 037c2269f - Update LocalSettings.php [19:14:29] [02mw-config] 07paladox synchronize pull request 03#2823: Remove EducationProgram extension - 13https://git.io/Je989 [19:14:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je98Q [19:14:47] [02miraheze/mw-config] 07paladox 034c3bbc8 - Update ManageWikiExtensions.php [19:14:48] [02mw-config] 07paladox synchronize pull request 03#2823: Remove EducationProgram extension - 13https://git.io/Je989 [19:14:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je987 [19:15:00] [02miraheze/mw-config] 07paladox 030243162 - Update extension-list [19:15:02] [02mw-config] 07paladox synchronize pull request 03#2823: Remove EducationProgram extension - 13https://git.io/Je989 [19:15:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:17:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:19:02] [02mw-config] 07paladox closed pull request 03#2823: Remove EducationProgram extension - 13https://git.io/Je989 [19:19:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Je98A [19:19:05] [02miraheze/mw-config] 07paladox 0317ee087 - Remove EducationProgram extension (#2823) * Remove EducationProgram extension Extension now unmaintained and does not support 1.34+ * Update LocalSettings.php * Update ManageWikiExtensions.php * Update extension-list [19:19:06] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:19:08] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:20:55] !log rebuild lc on mw* and lizardfs6 [19:21:53] !log rebuild lc on test1 [19:21:53] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:22:44] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:52] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [19:24:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [19:24:12] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:27] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:24:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [19:26:10] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [19:26:18] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:20] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [19:26:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [19:28:26] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:38] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19039 bytes in 7.231 second response time [19:28:41] !log temporarily undeleting adhdwiki [19:30:16] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:55] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:12] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [19:31:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [19:31:28] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.394 second response time [19:32:01] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [19:32:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:32:19] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19039 bytes in 0.871 second response time [19:33:04] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19039 bytes in 0.264 second response time [19:33:13] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [19:33:15] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [19:33:16] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.673 second response time [19:33:29] Hey Zppix [19:33:37] I didnt do it [19:33:38] :P [19:33:40] RhinosF1: [19:34:03] Zppix: nah, cant blame you today [19:34:07] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [19:34:15] RhinosF1: damn [19:34:22] * Zppix logs into mw1 [19:34:25] what about now [19:34:27] America is too far awsy to give me a cold Zppix [19:34:47] i dont know its bloody cold here [19:34:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:35:08] its 30F [19:35:26] Zppix: took a paracetomol and now really tired [19:35:36] RhinosF1: what is paracetomol? [19:36:13] Zppix: tylenol i think you lot call ot [19:36:14] It [19:37:14] RhinosF1: ah yers [19:37:21] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 5.658 second response time [19:37:30] RhinosF1: we call it tylenol or by its drug name ibuprofen [19:37:45] paladox: https://www.calpol.co.uk/our-products/calpol-sixplus-fastmelts [19:37:46] [ CALPOL® SIXPLUS™ Fastmelts ] - www.calpol.co.uk [19:37:50] Zppix: ^ [19:38:04] It does taste nice [19:38:07] Zppix: ibuprofen isnt paracetomol [19:38:09] but i haven't had it in years :P [19:38:17] Two seperate things [19:38:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [19:38:25] paladox: this didbt [19:38:41] One of few things i can take with my reflex [19:39:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [19:39:37] Hi JohnLewis [19:40:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:40:26] Hi [19:41:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:44:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [19:45:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [19:47:48] someone want to unstick both https://meta.miraheze.org/wiki/Special:GlobalRenameProgress [19:48:01] [ Global rename progress - Miraheze Meta ] - meta.miraheze.org [19:48:14] Voidwalker: i think paladox is working on apap [19:48:22] see -staff [19:48:40] just saw [19:49:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:50:13] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:50:48] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:23] !log re-delete adhdwiki [19:51:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:54:40] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 6.546 second response time [19:54:48] * hispano76 greeting JohnLewis Voidwalker :) [19:56:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:57:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je9BT [19:57:46] [02miraheze/puppet] 07paladox 03be5361e - mediawiki Increase job runners to 6 [19:59:12] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:26] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:30] !log restart nginx and php7.3-fpm on mw* [20:02:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:03:34] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 7.739 second response time [20:03:55] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 6.239 second response time [20:04:20] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 7.724 second response time [20:06:50] !log hacked redis config on misc2 to use volatile-lfu [20:07:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je9BC [20:07:24] [02miraheze/puppet] 07paladox 03e723d95 - Update jobrunner.json.erb [20:07:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:09:10] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [20:13:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:14:48] !log unhacked redis conifg file [20:15:20] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:15:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:16:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:18:21] hispano76: hey [20:22:19] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03DBConnRef [+0/-0/±1] 13https://git.io/Je9BD [20:22:21] [02miraheze/ManageWiki] 07paladox 0316bc206 - Fix "ManageWikiFormFactory::getFormDescriptor() must be an instance of Wikimedia\Rdbms\Database" [20:22:54] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-1/±1] 13https://git.io/Je9B9 [20:22:55] [02miraheze/mediawiki] 07paladox 0378ebe42 - Remove EducationProgram extension It's now unmaintained and no longer supported for 1.34+. [20:22:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:24:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:30:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [20:31:23] :) [20:32:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:35:20] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:35:40] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-udp-from-any-to-any-port-53] [20:36:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:36:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:37:41] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 7.195 second response time [20:38:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:38:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:39:35] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:42:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:43:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:43:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [20:45:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:45:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:46:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-1/±1] 13https://git.io/Je9Rn [20:46:12] [02miraheze/mediawiki] 07paladox 03a312f31 - Remove EducationProgram extension [20:46:47] [02miraheze/mediawiki] 07paladox pushed 037 commits to 03REL1_34 [+0/-0/±18] 13https://git.io/Je9RC [20:46:48] [02miraheze/mediawiki] 07lens0021 030f334b3 - Mark options as requiring parameters in addSite.php Bug: T239561 Change-Id: Ibd967da45f32c8ea58b8997f15d26ab06f1e14cb [20:46:50] [02miraheze/mediawiki] 07physikerwelt 03ff4d49a - Mimic CURLOPT_POST in GuzzleHttpRequest The MWHttpRequest is implemented by the CurlHttpRequest class and also the GuzzleHttpRequest class. However, curl based rendering set the CURLOPT_POST which implies that the 'Content-Type' header defaults to 'application/x-www-form-urlencoded'. To homgonize the functionality this patch mimics the curl behaviour in Guzzle. [20:46:50] Bug: T232866 Change-Id: Id60a8de18e5f1e750a3bde23bd8b0deca4071165 (cherry picked from commit 5e3a0e73955d6324c5dd6e12fbe36d3ba203d9db) [20:46:51] [02miraheze/mediawiki] 07reedy 03c59be32 - Update RELEASE-NOTES-1.34 Change-Id: Ie9c70854e759f848f6907b79b9ca630504958517 [20:46:53] [02miraheze/mediawiki] ... and 4 more commits. [20:48:57] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Je9Rl [20:48:59] [02miraheze/mediawiki] 07Ladsgroup 0329f232b - SECURITY: Do not allow user scripts on Special:PasswordReset Bug: T192134 Change-Id: If5e91452f2e569476626bcf650ba4efaa122952c [20:51:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [20:54:50] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Je9Ra [20:54:52] [02miraheze/mediawiki] 07paladox 030ddc274 - Update ManageWiki [20:55:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [20:55:43] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [20:59:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:03:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:26:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [21:32:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:33:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [21:41:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [21:47:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [21:49:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:13:05] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [22:13:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [22:21:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:21:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:35:59] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:38:05] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:41:10] !log root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=ucroniaswiki --update [22:41:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:47:17] paladox: Thanks Thanks :) [22:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:01:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:02:42] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:08] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:03:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [23:04:44] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 4.536 second response time [23:05:14] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 8.676 second response time [23:13:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:14:34] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Package[openssh-server],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [23:15:32] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:37] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:17:39] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 8.392 second response time [23:17:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:18:43] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.319 second response time [23:22:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:53] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:59] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19051 bytes in 1.545 second response time [23:27:14] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:36] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:28:42] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:32:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [23:34:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:34:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:35:45] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.497 second response time [23:36:03] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 9.734 second response time [23:39:13] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.529 second response time [23:39:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:45:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:46:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:47:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:48:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:53:14] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures