[00:00:54] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:00:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [00:01:03] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:01:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [00:03:05] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 2.729 second response time [00:03:07] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 2.629 second response time [00:03:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:04:18] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [00:05:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:21:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [01:23:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [01:30:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [01:31:48] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:50] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [01:32:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:33:48] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 2.479 second response time [01:33:49] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19053 bytes in 0.841 second response time [01:46:27] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1437 MB (5% inode=94%); [02:05:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [02:13:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:15:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-53] [03:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [03:53:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [04:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [04:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:33:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter] [04:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [04:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [05:03:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:13:12] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.45, 3.14, 2.51 [05:15:13] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.08, 3.10, 2.57 [05:43:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [05:51:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:26:28] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2319 MB (9% inode=94%); [06:43:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:51:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:03:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [07:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:42:47] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mrlove.wiki' expires in 15 day(s) (Mon 23 Dec 2019 07:40:41 AM GMT +0000). [07:43:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeyMX [07:43:02] [02miraheze/ssl] 07MirahezeSSLBot 03292b83b - Bot: Update SSL cert for mrlove.wiki [07:52:45] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Fri 06 Mar 2020 06:42:55 AM GMT +0000. [08:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeyMp [08:05:08] [02miraheze/services] 07MirahezeSSLBot 03c394e26 - BOT: Updating services config for wikis [08:15:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 4 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:21:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [09:10:18] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jeyyv [09:10:20] [02miraheze/dns] 07Reception123 037c5b9b9 - add baharna.org zone [09:23:01] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/Jeyym [09:23:02] [02miraheze/ssl] 07Reception123 03e1c0ebb - add www.baharna.org cert [09:31:03] Hey Reception123 [09:32:04] hi [09:32:28] Reception123: I’m going to do mailerlite about 12 [09:32:28] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:32:39] Then late tonight for training [09:32:41] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:32:45] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:32:48] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:33:09] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:33:10] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:33:42] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:33:47] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:34:40] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:39:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [09:41:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:16:50] !log created site ‘test1’ on mailerlite [10:16:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:31:18] @Operations: can someone help me add our dns to mailerlite [10:32:39] PuppyKun, Reception123: ^ [10:33:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [10:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeySU [10:35:08] [02miraheze/services] 07MirahezeSSLBot 038df7d7e - BOT: Updating services config for wikis [10:36:43] [02dns] 07RhinosF1 opened pull request 03#119: Add mailerlite TXT and SPF records to sign mail - 13https://git.io/JeyST [10:37:47] [02dns] 07RhinosF1 commented on pull request 03#119: Add mailerlite TXT and SPF records to sign mail - 13https://git.io/JeySI [10:41:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [10:46:57] RhinosF1: I'm not great with DNS but let me check [10:47:35] yeah I'd rather wait for someone who knows more about DNS [10:51:44] Reception123: k [10:52:19] !log starting deploy mailerlite - create operations accounts [10:52:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:53:36] !log adding owen, void and zppix as well (so making that everyone on staff@) [10:53:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-enable] [11:01:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [11:06:56] !log finished deploying mailerlite accounts [11:07:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:18:12] !log finished deploying mailerlite landing page 'test1' [11:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:35:43] !log finished deploy mailerlite (set up important and all email groups, configure workflow) [11:35:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:48:43] !log reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki inforevivalwiki /home/reception/InfoRevival20191207.xml (screen) [11:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:52:39] paladox, Reception123: please check https://phabricator.miraheze.org/T4536 subtasks and see if you can resolve [11:52:40] [ ⚓ T4536 Start Miraheze Mailing Lists ] - phabricator.miraheze.org [11:53:58] ok [11:59:50] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jey9o [11:59:52] [02miraheze/ssl] 07paladox 039421b2e - fix [11:59:59] paladox: check my dns pr [12:00:32] Reception123 ^ [12:01:25] paladox: could it really be that? [12:01:33] yup [12:02:52] Reception123 works now [12:03:06] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [12:03:21] Thanks! [12:03:22] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:03:22] RhinosF1 it looks fine to me, but rather john or SPF|Cloud look [12:03:31] ok [12:03:38] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [12:03:46] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:38] Reception123 puppet was failing judging from ^ [12:04:41] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:43] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:58] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [12:07:10] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [12:10:45] paladox: hm no idea why [12:23:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter] [12:33:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:39:11] paladox, Reception123: should we consider a separate tag to mail for the mailing lists or? [12:43:16] on phab [12:50:02] I guess so [12:51:12] paladox: can you create it? [12:51:38] hold on [12:51:42] oh [12:51:45] nvm [12:52:25] RhinosF1 https://phabricator.miraheze.org/project/view/49/ [12:52:25] [ Mailing-List · Manage ] - phabricator.miraheze.org [12:54:19] paladox: thx [12:55:54] paladox: see discord request as well [13:03:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [13:11:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:23:54] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [13:31:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:08:44] .ask JohnLewis can you merge https://github.com/miraheze/dns/pull/119 (T4536) [14:08:45] RhinosF1: I'll pass that on when JohnLewis is around. [14:08:47] [ Add mailerlite TXT and SPF records to sign mail by RhinosF1 · Pull Request #119 · miraheze/dns · GitHub ] - github.com [14:08:55] SPF|Cloud: ^ [14:13:54] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [14:29:13] Hey apap [14:29:31] hi [14:41:55] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:43:45] 🐶et [14:49:04] !log depool lizardfs6 [14:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:00:56] Hey, any Sysadmin around? [15:02:09] Hi [15:02:27] @Bitwise. how can I help? [15:02:47] So uhh, it appears that I lost my password and I have no email attached [15:02:58] and I would like to know if there's a way to be helped [15:03:14] @Bitwise. can you PM me your Username? [15:14:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:14:11] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:17] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:14:59] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:16:07] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:00] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:06] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.685 second response time [15:18:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [15:18:08] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 2.377 second response time [15:18:10] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 1.281 second response time [15:18:19] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.417 second response time [15:18:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [15:19:05] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [15:20:04] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [15:20:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [15:21:02] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19071 bytes in 0.273 second response time [15:21:26] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [15:22:40] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [15:23:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:23:37] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:24:37] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [15:26:18] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:27:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:27:51] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 7.704 second response time [15:28:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [15:28:27] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.395 second response time [15:30:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:31:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:42:31] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [15:43:10] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21978 MB (5% inode=96%); [15:45:15] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 42448 MB (11% inode=96%); [16:01:38] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:02:50] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:59] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 6.03, 10.91, 7.96 [16:03:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [16:03:42] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 50 minutes ago with 0 failures [16:04:58] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 1.937 second response time [16:05:25] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:06:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [16:07:29] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 4.018 second response time [16:09:26] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 12.85, 9.62, 6.64 [16:14:33] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:17:11] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:28] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:45] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:20] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 7.773 second response time [16:19:35] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 6.241 second response time [16:19:50] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 5.600 second response time [16:21:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:22:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:22:29] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:25:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [16:26:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:29:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:33:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:35:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:38:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:43:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [16:45:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:46:32] [02dns] 07Southparkfan commented on pull request 03#119: Add mailerlite TXT and SPF records to sign mail - 13https://git.io/Jey5b [16:49:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:50:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:52:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:52:26] [02dns] 07RhinosF1 commented on pull request 03#119: Add mailerlite TXT and SPF records to sign mail - 13https://git.io/JeydU [16:53:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:55:21] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:55:37] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:56:48] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:01:00] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [17:01:48] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:03:45] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [17:04:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 4.44, 5.13, 7.88 [17:05:05] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:57] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [17:07:12] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.60, 4.49, 7.18 [17:08:37] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.26, 4.09, 6.22 [17:09:12] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.11, 4.79, 6.69 [17:14:15] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:45:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [18:52:40] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [18:54:20] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:56:18] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 1.015 second response time [19:03:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:07:59] https://status.discordapp.com/ - Discord are having a few issues, I cant connect at the moment [19:07:59] [ Discord Status ] - status.discordapp.com [19:34:59] [02mw-config] 07GustaveLondon776 opened pull request 03#2822: Update LocalSettings.php - 13https://git.io/JeyAs [19:40:19] [02mw-config] 07RhinosF1 closed pull request 03#2822: Update LocalSettings.php - 13https://git.io/JeyAs [19:40:51] [02mw-config] 07RhinosF1 commented on pull request 03#2822: Update LocalSettings.php - 13https://git.io/JeyAc [19:41:46] ^ told before and said again [19:41:56] This is why training is needed [19:42:51] training?! [19:43:31] paladox: yes, on what is done via what [19:44:13] so who would be trained? [19:44:17] london [19:44:55] london? [19:44:55] or whoever who doesn't have training [19:45:25] paladox: GustaveLondon would be advised to have a look [19:45:27] gustave london. [19:46:48] RhinosF1 look at what? [19:47:41] paladox: some training when it’s expanded to non shell PR submitters [19:47:52] expanded?! [19:48:04] why will non shell be required to do it? [19:48:16] paladox: advised [19:48:29] ok, why advised too? [19:48:50] paladox: the number of pointless PRs thatll never be accepted [19:48:56] miraheze is happy to recieve any pulls from anyone without training [19:49:12] paladox: we can also help [19:49:21] Them make useful contribution [19:50:47] So any user who contributes, will be viewed as not doing it usefully till they do a training course? [19:50:55] paladox: no [19:51:17] If they clearly understand what theyre doing, go ahead and carry on [19:51:17] But that is what will happen. [19:51:49] But it won’t, I never said block anyone who didnt have training [19:52:03] I said if people are making mistakes, advise them how they can help [19:52:19] Via a course in mediawiki [19:52:24] Advise is different to pushing for them to go on a course [19:53:22] You know what will help people? Invite them to IRC or discord and activly help them/ [19:53:43] paladox: check their talk page on meta [19:53:53] so then a course won't help them [19:54:51] We can offer many ways they can helo [19:54:54] But having live conversations with people will help them. [19:54:59] s/helo/help [19:54:59] RhinosF1 meant to say: We can offer many ways they can help [19:55:13] paladox: ive offered to speak to them on irc [19:55:18] ok [19:55:26] then why do you think a course will help them? [19:55:37] I've seen you many times say on his pr to take it to the community. [19:55:45] Why do you think a course will change that? [19:56:44] paladox: because we can offer upfront advice before they get to the point of submitting the pr advertised on meta. Its an option. Not saying they will or have to use it [19:57:02] Yes, but pushing people towards it is wrong. [19:57:30] They should have the freedom to decide without being pushed. [19:57:32] Did I ever say that it was good? [19:57:44] RhinosF1 the training course? [19:57:45] We can advertise it as an OPTION [19:57:49] paladox: yrs [19:57:51] Yes [19:58:03] Ok and no? [19:59:15] paladox: no to what? [19:59:25] no to <+RhinosF1> Did I ever say that it was good? [20:01:18] It was a rhetorical question [20:01:23] oh [20:02:18] paladox: am i making sense? [20:03:03] yes as long as we are not going to be messaging each user to take the course. [20:03:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [20:08:41] paladox: we are not [20:08:47] ok [20:08:52] then it's fine with me [20:21:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [21:20:06] PROBLEM - pastport.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pastport.org' expires in 15 day(s) (Mon 23 Dec 2019 09:16:20 PM GMT +0000). [21:20:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeypW [21:20:22] [02miraheze/ssl] 07MirahezeSSLBot 0365e870d - Bot: Update SSL cert for pastport.org [21:27:26] ^ known about relay, services involving discord should improve within the hour [21:30:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeypa [21:30:09] [02miraheze/services] 07MirahezeSSLBot 03d9d755d - BOT: Updating services config for wikis [21:34:02] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Fri 06 Mar 2020 08:20:14 PM GMT +0000. [21:35:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ssh] [21:41:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [21:44:41] .tell Zppix reboot the relay, api is still down [21:44:42] RhinosF1: I'll pass that on when Zppix is around. [22:03:28] testing relay [22:03:52] nope not yet [22:03:54] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [22:08:10] .in 30 mins check discord relays [22:08:10] RhinosF1: Okay, I will set the reminder for: 2019-12-07 - 22:38:10GMT [22:11:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:23:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [22:31:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:37:13] test relay [22:38:11] RhinosF1: check discord relays [22:38:25] just done it [22:38:30] ZppixBot: it fails [22:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [23:01:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:04:05] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [23:06:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:12:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:12:30] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [23:14:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:14:28] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 1.426 second response time [23:21:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix]