[00:02:55] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.11, 7.64, 6.91 [00:04:49] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.83, 7.55, 7.06 [00:10:32] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.37, 7.81, 7.44 [00:15:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [00:16:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.46, 7.12, 7.38 [00:23:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:20] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.10, 7.94, 7.68 [00:26:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.58, 7.60, 7.60 [00:30:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.82, 7.51, 7.47 [00:34:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.17, 7.39, 7.53 [00:50:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.19, 7.04 [00:52:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.44, 7.34, 7.14 [01:02:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.76, 7.54, 7.25 [01:04:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.57, 7.88, 7.49 [01:06:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.03, 8.14, 7.69 [01:08:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.75, 7.76, 7.61 [01:10:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.90, 8.38, 7.89 [01:16:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.90, 7.99, 8.00 [01:20:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1wy [01:20:09] [02miraheze/services] 07MirahezeSSLBot 031e610fb - BOT: Updating services config for wikis [01:25:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [01:32:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.18, 7.53, 7.23 [01:33:05] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:34:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.73, 7.54, 7.29 [01:36:24] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.13, 7.91, 7.50 [01:36:25] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.57, 7.63, 7.49 [01:42:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.63, 7.92, 7.65 [01:44:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.62, 7.68, 7.66 [01:53:05] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [01:54:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.21, 7.71, 7.54 [01:56:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.40, 7.21, 7.39 [02:05:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [02:06:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.04, 6.18, 6.72 [02:13:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:19:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.27, 7.00, 6.81 [02:23:42] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.47, 8.87, 7.65 [02:39:04] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.35, 7.32, 7.85 [02:41:03] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.66, 8.13, 8.06 [02:48:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.54, 7.69, 7.96 [02:50:35] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.54, 7.86, 7.94 [02:52:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.52, 7.60, 7.82 [02:54:24] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.19, 7.89, 7.90 [02:56:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.57, 7.12, 7.60 [03:00:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.54, 7.72, 7.65 [03:02:32] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.22, 7.53, 7.60 [03:12:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.82, 6.04, 6.59 [03:24:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.98, 7.18, 6.84 [03:26:06] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443],Package[php7.3-apcu] [03:26:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.10, 6.22, 6.53 [04:02:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.58, 7.04, 6.29 [04:02:59] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [04:05:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [04:06:25] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.91, 6.07, 6.21 [04:13:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:49] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 4 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[git_pull_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443],Package[php7.3-redis] [04:35:02] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:52:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.63, 6.91, 6.24 [04:53:31] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:54:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.95, 5.92, 6.01 [05:03:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:05:36] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx] [05:25:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [05:43:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:01:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.65, 6.80, 6.26 [06:03:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.27, 6.42, 6.22 [06:05:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:11:38] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:12:59] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.29, 8.31, 7.21 [06:13:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:16:53] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.55, 7.93, 7.49 [06:20:46] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.09, 7.91, 7.65 [06:24:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [06:25:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [06:25:52] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [06:26:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:44:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.22, 7.16, 7.70 [06:56:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.45, 5.57, 6.42 [07:04:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.83, 7.04, 6.68 [07:16:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.95, 5.97, 6.47 [07:25:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [07:33:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:45:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [07:53:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:02:22] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:25:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [08:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:33] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.55, 7.11, 6.17 [09:08:24] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.13, 6.39, 6.22 [09:12:30] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.49, 7.03, 6.50 [09:13:08] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:13:33] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:15:12] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 4.605 second response time [09:15:35] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 3.326 second response time [09:18:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.47, 5.91, 6.34 [09:33:45] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[ufw-default-deny],Service[ufw],Service[prometheus-node-exporter],Service[nagios-nrpe-server] [09:35:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [09:53:05] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:54:16] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:05:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [10:13:02] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.16, 7.05, 6.18 [10:14:57] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.24, 6.05, 5.98 [10:23:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:35:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [10:43:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:54:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.66, 7.25, 6.67 [10:56:20] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.98, 6.23, 6.41 [11:35:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [11:53:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:22] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.39, 7.07, 6.45 [12:05:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [12:06:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.32, 6.22, 6.25 [12:23:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:24:47] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.38, 6.98, 6.60 [12:28:40] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.26, 6.70, 6.63 [12:42:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.35, 6.45, 5.82 [12:44:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.41, 6.65, 6.08 [13:09:52] Hello Kevin77300! If you have any questions, feel free to ask and someone should answer soon. [13:09:59] Hello everyone ! [13:10:26] I would like to embed the following OpenAgenda to my wiki which is zw.fontainebleau-avon.fr [13:10:34] What you recommend for that ? [13:10:36] [13:15:58] Hello RhinosF1 [13:16:39] Kevin77300: what's up [13:17:20] better than you I guess. I've read what you wrote on the Community Noticeboard. Hope that you'll be able to fight against your illness [13:18:58] Thanks, I've just seen your message about an Iframe [13:18:58] I've ask a question a few minutes before you logged in. I don't know if I can re-ask it or you can go through the looger to read it ? [13:19:02] Try https://www.mediawiki.org/wiki/Special:MyLanguage/Extension:Widgets [13:19:03] [ Extension:Widgets - MediaWiki ] - www.mediawiki.org [13:19:21] You can turn it on from Special:ManageWiki/extensions [13:19:29] Yep, I've installed it, but I don't understand how it works while reading the mw page on it [13:19:52] I don't get the sentence : "To add a widget to your MediaWiki installation, just create a page in the Widget: namespace and then use the {{#widget:...}} parser function to include it in the pages of the wiki. " [13:20:54] On widget:widgetname please the Iframe code [13:21:17] Then use {{#widget: widgetname}} on the page you want it to appear on [13:21:33] paladox, Reception123: ^ around to help while I eat [13:22:17] so for example I create this : https://zw.fontainebleau-avon.fr/wiki/Widget:OpenAgenda and I copy/past the iframe inside of it [13:22:19] [ Widget:OpenAgenda — Zero Waste Fontainebleau Avon ] - zw.fontainebleau-avon.fr [13:22:36] and use it as you said above [13:29:58] Unfortunately I'm mobile [13:31:47] Ok I think I might have embedded something, but the frame is calling as js. Should I add it in https://zw.fontainebleau-avon.fr/wiki/MediaWiki:Common.js ? [13:31:51] [ MediaWiki:Common.js — Zero Waste Fontainebleau Avon ] - zw.fontainebleau-avon.fr [13:40:38] Kevin77300: what page do you want the widget to show on [13:41:34] RhinosF1 : I've called it on the homepage https://zw.fontainebleau-avon.fr/wiki/Accueil . It is being displayed right now just above the map, but it does not load anything [13:41:36] [ Zero Waste Fontainebleau Avon ] - zw.fontainebleau-avon.fr [13:41:57] I've ve added mw.loader.load( 'https://openagenda.com/js/embed/cibulBodyWidget.js' ); in the Common.js [13:41:58] [ 404 Not Found ] - openagenda.com [13:43:11] But I am thinking of removing the [14:17:53] [ OpenAgenda ] - openagenda.com [14:18:32] Kevin77300: ok [14:19:17] RhinosF1 : But I need to embed javascript for this [14:19:19] Luckily I can read bits of French [14:19:34] Try your best [14:19:49] Does anything show in the developer console? [14:23:24] Content Security Policy: Les paramètres de la page ont empêché le chargement d’une ressource à https://openagenda.com/js/embed/cibulBodyWidget.js (« default-src »). [14:24:02] Which can be translated into : the parameters of the page prevented to load a resource in http://... [14:24:43] Kevin77300: looks like it's (intentionally) been blocked. You'll need to request a CSP exemption on Phabricator [14:24:50] Or get it whitelisting [14:25:16] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.91, 6.72, 6.41 [14:27:09] RhinosF1 : What is the process to request a CSP exemption in Phabricator or a whitelisting ? [14:27:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.99, 6.18, 6.25 [14:28:08] Kevin77300: create a ticket and ask for openagenda.com to be whitelisted [14:31:21] RhinosF1 : Whiwh form should I use ? Extension/Configuration change , bug, security issue ? [14:32:01] Kevin77300: configuration Change [14:32:16] RhinosF1 : Thanks [14:33:25] Kevin77300: link me to it when done [14:35:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [14:48:48] RhinosF1 : Here is the ticket https://phabricator.miraheze.org/T4944 [14:48:49] [ ⚓ T4944 Whitelist openagenda.com ] - phabricator.miraheze.org [14:53:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.61, 6.71, 6.38 [15:15:03] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [15:15:09] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.09, 6.50, 6.44 [15:23:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [15:25:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:27:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:29:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:35:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [15:43:12] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:03:37] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.25, 6.94, 6.35 [16:09:23] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.77, 6.78, 6.59 [16:13:13] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.73, 7.22, 6.81 [16:17:04] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.01, 6.77, 6.77 [16:22:50] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.12, 7.34, 6.93 [16:24:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.64, 7.59, 7.13 [16:25:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [16:32:41] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.23, 7.41, 7.09 [16:33:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:36] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.26, 6.99, 7.01 [16:36:17] !log run sudo -u www-data php cargoRecreateData.php --wiki rswiki --table Level on mw1 —Failed [16:36:17] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:36:30] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.55, 6.34, 6.74 [16:36:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:38:44] !log root@mw1:/srv/mediawiki/w/extensions/Cargo/maintenance# php cargoRecreateData.php --wiki rswiki --table Level [16:38:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:40:19] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [16:55:09] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.58, 7.29, 6.82 [16:58:57] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.11, 6.67, 6.74 [17:02:47] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.61, 7.15 [17:04:46] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.76, 7.25, 7.12 [17:10:38] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.14, 7.35, 7.16 [17:14:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.98, 7.41, 7.37 [17:14:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [17:14:53] ping test paladox [17:15:21] ups. ehm [17:15:22] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:37] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 7.60, 7.44 [17:17:24] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 6.338 second response time [17:17:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [17:18:34] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.66, 7.79, 7.55 [17:18:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:19:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:22:26] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.39, 7.65, 7.49 [17:24:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.43, 7.21, 7.37 [17:28:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.02, 6.06, 6.68 [17:30:12] hispano76: hi? [17:34:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.41, 7.09, 6.93 [17:35:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [17:36:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.03, 6.65, 6.79 [17:37:24] paladox: wgExtraInterlanguageLinkPrefixes then you can't add to ManageWiki? curiosity [17:38:13] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:38:29] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:43] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:40:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:40:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:40:39] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 0.970 second response time [17:41:10] hispano76: nope, that requires support in managewiki to support its syntax and because it uses namespaces, needs MWN support too [17:42:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:42:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:42:33] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.283 second response time [17:42:50] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 1.197 second response time [17:43:26] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:54:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 7.22, 6.90 [18:03:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:05:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:08:35] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.12, 7.05, 6.90 [18:10:31] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.99, 7.35, 7.05 [18:12:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.50, 7.80, 7.29 [18:15:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [18:16:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.71, 7.93, 7.61 [18:18:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.86, 7.63 [18:20:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 8.00, 7.97, 7.72 [18:22:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.24, 8.25, 7.90 [18:23:02] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:31:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:33:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:42:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.21, 7.70, 8.00 [18:44:27] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.44, 8.37, 8.22 [18:46:13] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:47:09] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:11] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 3.597 second response time [18:49:06] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19053 bytes in 0.429 second response time [18:52:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.35, 7.68, 7.93 [18:54:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.92, 8.46, 8.20 [19:02:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 7.43, 7.78 [19:04:22] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.02, 7.85, 7.88 [19:05:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[exim4],Service[postfix],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [19:13:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:16:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.21, 7.22, 7.76 [19:20:22] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.59, 8.27, 7.99 [19:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je15c [19:25:09] [02miraheze/services] 07MirahezeSSLBot 03aa412a9 - BOT: Updating services config for wikis [19:28:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.92, 7.85, 7.97 [19:30:24] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 8.19, 8.07 [19:32:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.69, 7.77, 7.92 [19:34:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.87, 8.12, 8.05 [19:36:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.11, 7.76, 7.94 [19:40:18] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.19, 7.54, 7.71 [19:42:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.42, 7.54, 7.68 [19:50:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.79, 7.81, 7.55 [19:52:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.87, 7.87, 7.64 [19:53:14] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:54:24] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.59, 8.27, 7.86 [19:55:21] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 7.423 second response time [19:56:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.00, 7.69, 7.72 [20:00:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.97, 8.06, 7.84 [20:02:22] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:03:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [20:04:26] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.947 second response time [20:07:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:08:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.35, 7.59, 7.84 [20:20:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.22, 7.04, 7.15 [20:22:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.30, 6.69, 6.96 [20:24:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.53, 6.32, 6.77 [20:36:21] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.40, 7.35, 6.77 [20:36:22] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:37:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:38:23] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.95, 7.17, 6.83 [20:39:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:42:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.43, 7.40, 7.01 [20:44:18] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.89, 6.74, 6.85 [20:44:33] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:46:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.80, 6.37, 6.69 [20:50:20] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.51, 7.26, 6.91 [20:51:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [20:52:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.68, 7.45, 7.06 [20:53:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:54:50] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:37] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 4 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [20:55:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:56:48] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.263 second response time [20:57:37] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 2.113 second response time [20:57:40] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:02:30] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 7.40, 7.12 [21:02:54] !log depool mw2 [21:03:01] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:03:32] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:03:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [21:03:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:04:24] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.81, 6.98, 7.04 [21:04:56] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.393 second response time [21:05:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:06:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.74, 6.33, 6.76 [21:16:13] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [21:20:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:21:17] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [21:23:19] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19053 bytes in 0.363 second response time [21:24:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:24:54] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:30:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [21:32:41] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:48:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:48:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:52:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:52:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:04:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [22:05:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [22:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1bY [22:05:08] [02miraheze/services] 07MirahezeSSLBot 03c784263 - BOT: Updating services config for wikis [22:06:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:13:01] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:13:39] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.51, 7.29, 6.90 [22:15:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je1bC [22:15:09] [02miraheze/services] 07MirahezeSSLBot 03ee9bbc3 - BOT: Updating services config for wikis [22:17:28] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.99, 6.59, 6.75 [22:22:47] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:38:57] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:46:25] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.30, 6.54, 6.05 [22:46:46] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 4 minutes ago with 3 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis],Service[rsyslog] [22:47:08] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:48:21] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.74, 5.99, 5.93 [22:56:27] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.48, 7.05, 6.64 [22:57:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [22:58:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [22:59:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:00:06] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:02:19] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.12, 7.39, 6.94 [23:03:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:04:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:05:19] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [23:06:33] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.12, 7.58, 7.28 [23:08:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:08:32] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:12:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:12:44] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [23:13:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:14:02] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:14:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:16:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:26:41] !log repool mw2 [23:26:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:30:20] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.51, 6.51, 6.79 [23:34:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.51, 6.85, 6.86 [23:38:18] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.54, 6.04, 6.53