[00:03:42] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [00:04:48] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:11] paladox: After protecting the template, I get this message about a module. How can I fix it? Because it's exhausting to install as many modules as there are calls it makes. https://commons.miraheze.org/wiki/Template:Information [00:05:22] [ Template:Information - Miraheze Commons ] - commons.miraheze.org [00:05:38] you need to import that module [00:06:13] hispano76: Lua modules do amazing things, but they are stressful and annoying to deal with when importing, it's unfortunate :( [00:06:14] It's just that one module makes a call to another that makes a call to another. [00:08:20] :( [00:12:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [00:13:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:14:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:23:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [00:35:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7ct [00:35:09] [02miraheze/services] 07MirahezeSSLBot 03d961e18 - BOT: Updating services config for wikis [00:43:46] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:17] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1432 MB (5% inode=94%); [00:54:17] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1461 MB (6% inode=94%); [01:05:46] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [01:13:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:43] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1446 MB (5% inode=94%); [01:45:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [01:53:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:53:56] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [03:11:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [03:56:06] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[git_checkout_mathoid],Exec[git_checkout_3d2png],Exec[git_pull_MediaWiki config],Exec[git_checkout_MediaWiki core] [04:04:27] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:13:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [04:21:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [04:53:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [05:03:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:06:02] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [05:12:09] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:23:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [05:31:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:03:49] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Exec[ufw-allow-udp-from-any-to-any-port-53] [06:11:46] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:25:28] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1819 MB (7% inode=94%); [07:03:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:11:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:13:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:21:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [09:21:05] Hello wm-bot550! If you have any questions, feel free to ask and someone should answer soon. [09:31:49] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [09:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix] [09:51:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [10:13:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [10:41:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [12:03:46] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [12:21:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [12:25:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7rP [12:25:09] [02miraheze/services] 07MirahezeSSLBot 0381e98c3 - BOT: Updating services config for wikis [12:32:18] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:32:30] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:32:34] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:32:44] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:32:55] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:00] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:00] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:01] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:02] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:09] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:14] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:27] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:27] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:36] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:40] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:56] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:42:44] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [12:42:56] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [12:43:02] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [12:43:13] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [12:43:29] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [12:43:36] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:43:40] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [12:43:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:43:59] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:18] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:30] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:33] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:57] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:00] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:01] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:08] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [12:45:35] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:53:46] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [13:11:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [14:01:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:07:12] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je763 [14:07:14] [02miraheze/landing] 07paladox 032493169 - Update donation link [14:53:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [14:55:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7iM [14:55:09] [02miraheze/services] 07MirahezeSSLBot 03266748a - BOT: Updating services config for wikis [15:01:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:25:18] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:33:21] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:03:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:05:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:05:18] hey JohnLewis [16:07:41] Hey [16:08:22] how are you today JohnLewis ? [16:08:37] I'm good, you? [16:09:16] JohnLewis: absolutely ecstatic, I get 3 extra days off school [16:10:08] lucky :P [16:10:45] JohnLewis: very lucky for me seen I've escaped with only a cold. Not for those with norovirus (see PM) [16:11:21] Even more lucky for you then haha [16:11:48] JohnLewis: yep!!!! [16:19:23] !log update mailerlite donate link on 'test1' per metawiki [16:21:05] JohnLewis: a LogBot is dead and b) should someone kill my access to that. We said it needed to be done by end of month [16:22:01] Let me restart the bot and then we'll start on killing the last of your access [16:22:54] !log restarted logbot [16:23:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:23:24] !log 16:19:23 !log update mailerlite donate link on 'test1' per metawiki [16:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:24:16] Wait, do I even have access to mailerlite? :? [16:24:36] JohnLewis: you should have an email link to gain access [16:25:59] now I do :) [16:26:43] JohnLewis: there's that, eliademy, and then the mail backup to sort [16:27:13] MailerLite is gone now [16:29:35] JohnLewis: i'll look at eliademy and then you sort mail [16:32:31] .isup https://eliademy.com/ [16:34:40] https://eliademy.com/ looks down from here. [16:37:00] sent mail [16:41:51] JohnLewis: once eliademy works everything should be fine [16:43:57] Okay [16:54:22] !log hack redis config [16:55:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:56:05] hey Zppix [16:57:07] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:57:13] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [16:57:37] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [16:59:07] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24668 bytes in 0.811 second response time [16:59:34] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [17:45:04] * hispano76 greeting [17:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7DK [17:55:08] [02miraheze/services] 07MirahezeSSLBot 0325f5ab6 - BOT: Updating services config for wikis [18:02:24] Hello! I have a problem; I've fogotten my password, I don't have an email registered on my account, but I'm still logged in. [18:02:52] paladox: https://phabricator.miraheze.org/T4993#94961 [18:02:53] [ ⚓ T4993 Namespace removal request ] - phabricator.miraheze.org [18:03:14] Zamujasa: Hello! [18:03:40] Zamujasa, if your still logged in you can set email by going to Special:Preferences [18:03:53] That requires re-entering the password, though. [18:04:37] Zamujasa: do you have anyway for us to verify that is infact your account? [18:06:41] are you still able to edit Zamujasa ? [18:07:11] I can edit, I think, yes. I'm working on the identity thing right now, one moment [18:07:17] ok [18:08:27] answered hispano76 [18:09:20] thanks paladox :) [18:09:24] Zamujasa: please let me know when you have done that [18:10:54] The primary wiki I edit on is https://sol.miraheze.org/wiki/User:Zamujasa - I am a coder for the game server that wiki is about, and here is a post on that forum I made just now: https://forum.ss13.co/showthread.php?tid=12741&pid=142842#pid142842 [18:10:56] [ Permission error - Solarium Wiki ] - sol.miraheze.org [18:10:57] [ a thread to post random updates i make ] - forum.ss13.co [18:13:48] hi Zamujasa can you message me ( type /query Zppix) with the email you wish to have set to your account [18:18:07] !log run changePassword.php for a user that lost their login, temp password sent directly to user. (ON MW1) [18:18:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:19:18] Thanks for the help. Have a good one! [18:21:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:41:48] PuppyKun: you hold the wiki creator right but it’s on the steward group [18:42:11] Just letting you know as I spotted it tidying a few thing up if you want to remove it [18:44:24] Voidwalker: also if https://meta.miraheze.org/wiki/Special:Contributions/Proxybot is dead does it still need rights? [18:44:29] [ User contributions for Proxybot - Miraheze Meta ] - meta.miraheze.org [18:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7yK [18:45:09] uh, could ask Zppix what's the deal with it [18:45:10] [02miraheze/services] 07MirahezeSSLBot 03d5591f9 - BOT: Updating services config for wikis [18:46:03] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Package[php7.3-redis],Service[rsyslog],Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members] [18:46:41] Zppix: ? [18:46:44] I plan on reviving it eventually SantaRhino [18:47:10] Zppix: k, i was just cleaning things up [18:51:03] he's doing more work now he's gone than he did when he was here ;) [18:53:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7yd [18:53:31] [02miraheze/puppet] 07paladox 033a55244 - redis: Change maxmemory_policy to volatile-lfu [18:53:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [18:54:50] JohnLewis: heh, im mobile now [18:55:02] SantaRhino: no excuse :P [18:55:18] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:55:24] Can’t sysadmin mobile and my best time to be laptop is when i get the most impact of the day JohnLewis [19:02:42] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:08:42] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+2/-0/±1] 13https://git.io/Je7Sm [19:08:44] [02miraheze/mediawiki] 07paladox 031b74f37 - Install ContributionTracking and DonationInterface [19:09:45] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+2/-0/±1] 13https://git.io/Je7SO [19:09:46] [02miraheze/mediawiki] 07paladox 035a1c7b7 - Install ContributionTracking and DonationInterface [19:10:38] paladox: should those have .git on the end [19:10:45] no? [19:10:58] paladox: then fix it :) [19:11:22] oh, i see what you mean. It dosen't matter either way. [19:11:26] I copied the link from github [19:11:31] on gerrit it dosen't [19:11:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:12:12] paladox: It looks better anyway for everything to be the same format [19:12:56] contributions always welcome :) [19:13:53] paladox: 18:54:50 JohnLewis: heh, im mobile now [19:14:04] github provide an inline editor [19:14:09] Still am but i can fix tommorow [19:14:10] i always use that even when on mobile [19:14:19] GitHub Mobile is horrod [19:14:22] Horrid [19:14:40] +1 that comment [19:15:15] Coding is one of the few things I don’t like doing on a mobile [19:15:41] Is it? [19:15:44] it's fine for me :P [19:16:08] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+2/-0/±1] 13https://git.io/Je7SZ [19:16:09] [02miraheze/mediawiki] 07paladox 03a67dc24 - Install ContributionTracking and DonationInterface [19:16:11] paladox: what devoce you have [19:17:27] iphone [19:17:28] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+2/-0/±1] 13https://git.io/Je7SW [19:17:29] [02miraheze/mediawiki] 07paladox 033818e9a - Install ContributionTracking and DonationInterface [19:17:34] iPhone x to be precise [19:17:43] PROBLEM - wiki.2035.rocks - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.2035.rocks' expires in 15 day(s) (Thu 02 Jan 2020 07:14:02 PM GMT +0000). [19:17:48] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Je7S8 [19:17:50] [02miraheze/mediawiki] 07paladox 03b0c9999 - Fix [19:17:54] paladox: ah probably helps having bigger phone [19:17:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7SB [19:17:59] [02miraheze/ssl] 07MirahezeSSLBot 030a1e532 - Bot: Update SSL cert for wiki.2035.rocks [19:18:03] yup [19:18:26] paladox: any thougts on https://phabricator.wikimedia.org/T240885 [19:18:27] [ ⚓ T240885 Create a 2020+ plan for ZppixBot ] - phabricator.wikimedia.org [19:18:34] no [19:18:38] Im sure you’re on .botadmins [19:18:40] i doin't deal with that bot [19:18:44] So thought i’d ask [19:18:47] .botadmins [19:18:47] SantaRhino: The bot's admins are: SwisterTwister, Reception123, Reception|away, paladox, Zppix and Voidwalker [19:18:50] Yep [19:19:12] If anyone does please comment so me and Zppix can argue it out latet [19:20:34] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:21:00] i have an x too [19:21:59] :P [19:22:10] i'm upgrading for christmas to an even bigger phone [19:22:17] the iPhone 11 Max Pro [19:23:16] Lucky [19:23:43] RECOVERY - wiki.2035.rocks - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.2035.rocks' will expire on Mon 16 Mar 2020 06:17:50 PM GMT +0000. [19:23:45] lucky [19:24:40] SantaRhino: presumably if I ever lose steward I could keep wiki creator. [19:26:22] PuppyKun: dependent on how you got it [19:26:31] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:40] If you got Wiki Creator before steward, yes [19:26:47] I think [19:27:16] JohnLewis: that’s the rule right? [19:27:41] point was they're separate roles. Stewards and sysadmins overlap a lot you don't just remove one. steward overlaps a lot with local sysop/crat anywhere [19:27:44] PuppyKun: if you lose steward i think we have more issues [19:27:45] There's no rule regarding it [19:27:54] Zppix: I'm surprised I'm still a steward [19:28:06] PuppyKun: i mean you are technically active [19:28:15] PuppyKun: me too :? I mean why do we bother having you ;) [19:28:25] lol John [19:28:33] Zppix: not really? i mean a little on discord but meh [19:28:42] the note on my profile has been there for over a year lol [19:28:47] hey activity is activity no matter how small PuppyKun [19:29:19] Zppix: how do i set a reminder [19:29:21] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:29:45] PuppyKun: if you want to do a specfic time its .at if you want to do by a certain timeframe (like days) use .in [19:29:54] .help at [19:29:55] .help in [19:29:56] Zppix: The documentation for this command is too long; I'm sending it to you in a private message. [19:29:59] Zppix: Give user a reminder in the given amount of time. [19:30:00] Zppix: e.g. .in 3h45m Release a new version of ZppixBot [19:30:47] .in 1h30m nda [19:30:48] PuppyKun: Okay, I will set the reminder for: 2019-12-17 - 21:00:48UTC [19:31:00] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:43] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:32:58] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:33:21] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:22] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:33:37] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:00] PuppyKun: if you want you can set your timezone see .help settz [19:40:14] !log apt-get upgrade on lizardfs6 [19:40:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [19:51:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:03:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [20:11:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:23:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [20:31:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [20:43:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [20:51:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:51:54] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [20:52:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [20:53:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:56:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:00:49] PuppyKun: nda [21:17:46] paladox: 502 [21:17:53] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:18:02] that's a timeout [21:18:05] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:06] +1 [21:18:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:20:12] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 0.335 second response time [21:20:27] paladox: i know what it is [21:20:35] I wanted to edit [21:20:36] ok [21:22:53] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:23:37] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:24:17] lol SantaRhino [21:24:56] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [21:25:35] paladox: slow but through [21:25:38] Zppix: ? [21:25:45] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19054 bytes in 3.601 second response time [21:25:57] your messsage you sent 5 minutes ago [21:26:34] Zppix: replying to paladox’s reply [21:26:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:27:46] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:28:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [21:30:05] !log restart php7.3-fpm and nginx on mw* [21:30:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:30:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:31:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [21:32:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:33:48] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03DBConnRef [+0/-0/±1] 13https://git.io/Je7Qb [21:33:49] [02miraheze/ManageWiki] 07JohnFLewis 038fff382 - the -> this in some contexts [21:35:36] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je7Qx [21:35:37] [02miraheze/ManageWiki] 07JohnFLewis 039a7e233 - the -> this in some contexts [21:36:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:37:59] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Je77v [21:38:00] [02miraheze/mediawiki] 07JohnFLewis 035f28a9f - update MW [21:38:56] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:41:10] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:16] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [21:43:06] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [21:43:20] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 7.471 second response time [21:44:01] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [21:44:53] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.293 second response time [21:45:07] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 4.535 second response time [21:45:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:46:11] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.682 second response time [21:48:53] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:04] !log apt-get upgrade on mw* [21:49:20] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.390 second response time [21:49:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [21:49:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:50:12] !log reboot mw2 [21:51:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:51:21] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:19] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.481 second response time [21:52:42] !log reboot mw3 [21:52:44] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:53:22] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:54:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:54:08] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [21:54:17] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.672 second response time [21:54:36] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:54:47] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:55:28] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [21:55:47] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.004 second response time [21:56:37] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.659 second response time [21:56:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:56:55] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:03:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [22:11:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:28:31] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [22:28:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je75Z [22:28:54] [02miraheze/puppet] 07paladox 0376c3862 - varnish: Add stripe to csp whitelist [22:47:59] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je75j [22:48:00] [02miraheze/ManageWiki] 07JohnFLewis 03cdffac3 - fix synatx [22:48:22] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Je7de [22:48:24] [02miraheze/mediawiki] 07JohnFLewis 039062b51 - MW [22:53:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:02:23] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:03:21] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:19] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:05:40] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 9.802 second response time [23:07:45] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:12] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [23:08:38] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.006 second response time [23:08:42] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:09:45] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19059 bytes in 1.630 second response time [23:10:34] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [23:12:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:14:36] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:48] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [23:15:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:15:42] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1365 bytes in 0.036 second response time [23:16:42] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.004 second response time [23:17:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [23:17:49] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19064 bytes in 6.618 second response time [23:19:14] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [23:20:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:20:54] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:27] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:52] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:22:38] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:22:41] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:20] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:23:36] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 8.017 second response time [23:23:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [23:23:55] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 2.536 second response time [23:24:41] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19066 bytes in 0.358 second response time [23:25:01] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24668 bytes in 0.726 second response time [23:27:14] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [23:27:59] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:28:31] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [23:29:13] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:29:56] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24668 bytes in 0.003 second response time [23:30:36]