[00:35:53] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeAzs [00:35:54] [02miraheze/MirahezeMagic] 07paladox 039c23b14 - Add "Recent Changes" i18n [00:35:56] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX [00:35:57] [02MirahezeMagic] 07paladox opened pull request 03#93: Add "Recent Changes" i18n - 13https://git.io/JeAzG [00:38:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeAzn [00:38:46] [02miraheze/mw-config] 07paladox 03ac2ca03 - Move "Recent Changes" configs to it's own section [00:38:47] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:38:49] [02mw-config] 07paladox opened pull request 03#2843: Move "Recent Changes" configs to it's own section - 13https://git.io/JeAzc [00:39:05] [02MirahezeMagic] 07paladox closed pull request 03#93: Add "Recent Changes" i18n - 13https://git.io/JeAzG [00:39:06] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAzC [00:39:08] [02miraheze/MirahezeMagic] 07paladox 03e9b8fd7 - Add "Recent Changes" i18n (#93) [00:40:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JeAzW [00:40:06] [02miraheze/mediawiki] 07paladox 037d6b4fe - Update MirahezeMagic [00:40:19] [02mw-config] 07paladox edited pull request 03#2843: Move "Recent Changes" configs to it's own section - 13https://git.io/JeAzc [00:41:15] [02mw-config] 07paladox closed pull request 03#2843: Move "Recent Changes" configs to it's own section - 13https://git.io/JeAzc [00:41:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAzB [00:41:18] [02miraheze/mw-config] 07paladox 0363c0121 - Move "Recent Changes" configs to it's own section (#2843) [00:41:19] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:41:21] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:42:54] !log rebuild lc on mw* and lizardfs6 [00:44:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:46:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:47:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:52:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:54:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:54:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:55:05] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:58:20] !log root@misc2:/srv/redis# rm appendonly.aof [00:58:28] !log root@misc2:/srv/redis# rm dump.rdb [00:58:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:00:20] !log test [01:00:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:01:17] !log restart logbot on misc1 [01:01:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:01:35] !log test [01:01:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:03:22] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [01:08:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [01:10:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:03:50] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [03:03:51] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown) [03:03:59] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 2 minutes ago with 15 failures. Failed resources (up to 3 shown) [03:04:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [03:04:12] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 48 failures. Last run 2 minutes ago with 48 failures. Failed resources (up to 3 shown) [03:04:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [03:04:39] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown) [03:04:44] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [03:04:48] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 422 failures. Last run 3 minutes ago with 422 failures. Failed resources (up to 3 shown) [03:04:52] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 232 failures. Last run 2 minutes ago with 232 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades] [03:05:04] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown) [03:05:09] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 427 failures. Last run 3 minutes ago with 427 failures. Failed resources (up to 3 shown) [03:05:34] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 421 failures. Last run 3 minutes ago with 421 failures. Failed resources (up to 3 shown): File[/usr/lib/php/20180731/tideways_xhprof.so],File[/etc/systemd/system/jobrunner.service],File[/etc/systemd/system/jobchron.service],File[/etc/rsyslog.d/20-jobrunner.conf] [03:05:35] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 416 failures. Last run 3 minutes ago with 416 failures. Failed resources (up to 3 shown) [03:06:06] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 421 failures. Last run 3 minutes ago with 421 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org_private],File[/opt/ploticus_2.42-3+b4_amd64.deb],File[/opt/texvc_3.0.0+git20160613-1_amd64.deb],File[/etc/mathoid/config.yaml] [03:06:25] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 122 failures. Last run 2 minutes ago with 122 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org_private],File[/etc/default/stunnel4],File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends] [03:06:48] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 3 minutes ago with 16 failures. Failed resources (up to 3 shown): File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common] [03:12:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [03:12:40] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [03:12:50] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:13:03] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:13:25] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:13:32] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [03:13:45] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:13:46] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:13:49] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:21] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:14:23] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:14:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:14:52] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:15:22] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:15:29] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:16:21] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:30:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeA2i [03:30:09] [02miraheze/services] 07MirahezeSSLBot 03508464c - BOT: Updating services config for wikis [05:13:40] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [05:14:22] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 47 failures. Last run 2 minutes ago with 47 failures. Failed resources (up to 3 shown) [05:14:48] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown) [05:14:52] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [05:14:53] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [05:15:00] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [05:15:08] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 427 failures. Last run 3 minutes ago with 427 failures. Failed resources (up to 3 shown) [05:15:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [05:15:16] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 422 failures. Last run 3 minutes ago with 422 failures. Failed resources (up to 3 shown) [05:15:28] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [05:15:45] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 424 failures. Last run 3 minutes ago with 424 failures. Failed resources (up to 3 shown): File[/etc/mathoid/config.yaml],File[/etc/apt/trusted.gpg.d/php.gpg],File[/usr/lib/php/20180731/luasandbox.so],File[/usr/lib/php/20170718/luasandbox.so] [05:15:47] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 422 failures. Last run 3 minutes ago with 422 failures. Failed resources (up to 3 shown) [05:15:48] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 412 failures. Last run 4 minutes ago with 412 failures. Failed resources (up to 3 shown) [05:15:54] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 4 minutes ago with 9 failures. Failed resources (up to 3 shown) [05:15:56] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 431 failures. Last run 3 minutes ago with 431 failures. Failed resources (up to 3 shown): File[/usr/local/bin/gen_fingerprints],File[/etc/vim/vimrc.local],File[/usr/share/nginx/favicons/default.ico],File[/usr/share/nginx/favicons/apple-touch-icon-default.png] [05:16:23] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 137 failures. Last run 2 minutes ago with 137 failures. Failed resources (up to 3 shown): File[/etc/default/stunnel4],File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate] [05:16:47] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 208 failures. Last run 3 minutes ago with 208 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [05:22:42] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [05:22:58] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:23:05] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:23:28] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:23:36] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [05:23:39] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:23:41] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [05:23:52] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:23:53] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:24:17] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:24:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:24:39] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:24:42] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:24:44] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:25:18] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:25:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:26:22] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:24:38] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 43 failures. Last run 2 minutes ago with 43 failures. Failed resources (up to 3 shown) [06:24:44] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [06:24:44] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [06:24:45] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown) [06:25:00] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown) [06:25:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [06:25:31] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 418 failures. Last run 3 minutes ago with 418 failures. Failed resources (up to 3 shown) [06:25:34] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 423 failures. Last run 3 minutes ago with 423 failures. Failed resources (up to 3 shown) [06:25:44] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 420 failures. Last run 3 minutes ago with 420 failures. Failed resources (up to 3 shown): File[/usr/lib/php/20180731/tideways_xhprof.so],File[/etc/ImageMagick-6/policy.xml],File[/srv/mediawiki/robots.php],File[/usr/local/bin/fileLockScript.sh] [06:25:48] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 420 failures. Last run 3 minutes ago with 420 failures. Failed resources (up to 3 shown) [06:25:49] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 4 minutes ago with 6 failures. Failed resources (up to 3 shown) [06:25:50] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 407 failures. Last run 3 minutes ago with 407 failures. Failed resources (up to 3 shown) [06:25:51] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 4 minutes ago with 7 failures. Failed resources (up to 3 shown) [06:25:52] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 4 minutes ago with 5 failures. Failed resources (up to 3 shown) [06:26:02] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3249 MB (13% inode=94%); [06:26:07] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:26:21] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 134 failures. Last run 2 minutes ago with 134 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [06:26:44] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 204 failures. Last run 3 minutes ago with 204 failures. Failed resources (up to 3 shown): File[/etc/update-motd.d/97-last-puppet-run],File[/etc/sudoers],File[/home/nagiosre],File[/etc/nginx/mime.types] [06:32:58] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [06:33:03] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:33:47] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:33:48] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:33:48] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:33:51] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:33:58] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:34:00] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:01] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:26] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:34:33] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:34:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:36] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:37] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:37] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:35:28] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:36:14] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:49] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeAo1 [06:47:50] [02miraheze/ManageWiki] 07translatewiki 0320e3b1e - Localisation updates from https://translatewiki.net. [06:47:51] [ Main page - translatewiki.net ] - translatewiki.net. [06:47:52] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JeAoM [06:47:54] [02miraheze/MirahezeMagic] 07translatewiki 032926a40 - Localisation updates from https://translatewiki.net. [06:47:55] [ Main page - translatewiki.net ] - translatewiki.net. [08:53:48] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown) [08:54:42] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 2 minutes ago with 40 failures. Failed resources (up to 3 shown) [08:54:48] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [08:54:50] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown) [08:54:51] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown) [08:54:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [08:54:52] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [08:54:52] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [08:54:59] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [08:55:28] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 419 failures. Last run 3 minutes ago with 419 failures. Failed resources (up to 3 shown) [08:55:32] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 418 failures. Last run 3 minutes ago with 418 failures. Failed resources (up to 3 shown) [08:55:42] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 419 failures. Last run 3 minutes ago with 419 failures. Failed resources (up to 3 shown): File[/usr/lib/php/20180731/tideways_xhprof.so],File[/etc/ImageMagick-6/policy.xml],File[/srv/mediawiki/robots.php],File[/usr/local/bin/fileLockScript.sh] [08:55:45] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 416 failures. Last run 3 minutes ago with 416 failures. Failed resources (up to 3 shown) [08:55:55] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 417 failures. Last run 3 minutes ago with 417 failures. Failed resources (up to 3 shown): File[/usr/lib/php/20180731/tideways_xhprof.so],File[/etc/ImageMagick-6/policy.xml],File[/srv/mediawiki/robots.php],File[/usr/local/bin/fileLockScript.sh] [08:55:56] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 408 failures. Last run 4 minutes ago with 408 failures. Failed resources (up to 3 shown) [08:56:17] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 125 failures. Last run 2 minutes ago with 125 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [08:56:33] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 199 failures. Last run 3 minutes ago with 199 failures. Failed resources (up to 3 shown): File[/etc/sudoers],File[/home/nagiosre],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params] [09:02:47] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:02:55] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:02:56] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:02:58] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [09:03:33] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [09:03:38] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [09:03:40] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:03:49] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:03:53] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:18] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [09:04:32] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:04:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:38] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:39] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:04:39] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:06:01] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:06:16] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:13:01] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:13:06] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:13:20] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 2 minutes ago with 16 failures. Failed resources (up to 3 shown) [10:13:21] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:13:27] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:13:35] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 440 failures. Last run 2 minutes ago with 440 failures. Failed resources (up to 3 shown) [10:13:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:13:39] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:13:45] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [10:14:12] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 3 minutes ago with 51 failures. Failed resources (up to 3 shown) [10:14:17] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:14:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:14:38] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 3 minutes ago with 29 failures. Failed resources (up to 3 shown) [10:14:39] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 194 failures. Last run 3 minutes ago with 194 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [10:14:39] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [10:14:44] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [10:15:35] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 4 minutes ago with 246 failures. Failed resources (up to 3 shown) [10:22:58] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [10:23:03] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [10:23:20] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:23:36] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [10:23:48] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [10:23:48] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:24:08] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:24:32] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [10:24:34] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:24:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:24:37] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:24:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:25:15] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:25:22] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:25:28] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:25:44] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:26:15] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:48:58] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.03, 2.30, 1.64 [10:50:57] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.18, 1.14, 1.32 [10:55:00] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.52, 3.16, 2.27 [10:58:57] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.87, 1.71, 1.94 [11:00:56] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.75, 1.19, 1.68 [11:43:23] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:43:31] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 49 failures. Last run 2 minutes ago with 49 failures. Failed resources (up to 3 shown) [11:43:34] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:43:40] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 2 minutes ago with 13 failures. Failed resources (up to 3 shown) [11:44:18] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 122 failures. Last run 2 minutes ago with 122 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [11:44:33] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 190 failures. Last run 2 minutes ago with 190 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades] [11:44:35] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 3 minutes ago with 27 failures. Failed resources (up to 3 shown) [11:44:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:44:37] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown) [11:44:41] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown) [11:44:53] paladox: ^ [11:44:59] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 22 failures. Last run 3 minutes ago with 22 failures. Failed resources (up to 3 shown) [11:45:00] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 3 minutes ago with 29 failures. Failed resources (up to 3 shown) [11:45:14] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 239 failures. Last run 3 minutes ago with 239 failures. Failed resources (up to 3 shown) [11:45:25] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 4 minutes ago with 15 failures. Failed resources (up to 3 shown) [11:45:26] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 245 failures. Last run 3 minutes ago with 245 failures. Failed resources (up to 3 shown) [11:45:31] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 438 failures. Last run 3 minutes ago with 438 failures. Failed resources (up to 3 shown) [11:45:35] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 239 failures. Last run 3 minutes ago with 239 failures. Failed resources (up to 3 shown) [11:52:37] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [11:52:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [11:53:02] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [11:53:03] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [11:53:18] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [11:53:25] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [11:53:29] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [11:53:33] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [11:53:36] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [11:53:43] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [11:53:44] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:53:47] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [11:54:32] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [11:54:34] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:54:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:55:22] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:56:15] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:43:34] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.80, 3.02, 1.88 [13:02:58] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:03:14] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:03:21] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:03:22] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:03:22] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:03:32] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 242 failures. Last run 2 minutes ago with 242 failures. Failed resources (up to 3 shown) [13:03:40] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 2 minutes ago with 51 failures. Failed resources (up to 3 shown) [13:03:41] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:03:42] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 2 minutes ago with 15 failures. Failed resources (up to 3 shown) [13:03:46] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.02, 1.35, 1.93 [13:04:07] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [13:04:18] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:04:35] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:04:37] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 3 minutes ago with 29 failures. Failed resources (up to 3 shown) [13:04:37] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:04:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:04:39] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [13:04:41] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:05:45] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 6.43, 2.89, 2.31 [13:11:48] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.58, 1.53, 1.96 [13:12:41] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:12:58] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:13:11] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:13:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:13:40] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:13:48] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.53, 2.21, 2.13 [13:14:07] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:14:31] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:14:32] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:14:34] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:14:38] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:14:39] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:15:15] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:15:22] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:15:23] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:15:25] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:15:33] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:16:16] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:19:47] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.45, 1.15, 1.75 [13:21:45] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.90, 0.96, 1.54 [13:37:14] !log reboot puppet1 [13:37:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:39:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JeADK [14:39:12] [02miraheze/mediawiki] 07paladox 03acb34b4 - Update LinkTitles [15:14:46] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [15:16:30] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [15:16:53] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [15:18:27] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [15:43:11] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.32, 2.77, 1.93 [15:43:48] Hi hispano76 [15:45:11] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.79, 1.70, 1.68 [15:45:20] hi RhinosF1 :) [15:45:47] hispano76: doing much for NYE? [15:48:41] NYE? [15:48:47] RhinosF1: hi [15:48:50] hispano76: New Year's Eve [15:48:50] hispano76: hi [15:49:02] RhinosF1: doing a Phabricator cleanup so we start 2020 with a smaller backlog :) [15:50:31] RhinosF1: Not really. XD [15:50:35] hi Reception123 [15:54:41] I mean, not really much.* [15:56:03] Reception123: i saw [16:15:08] RhinosF1: https://phabricator.miraheze.org/T5014#95437 would you mind doing a PR? [16:15:10] [ ⚓ T5014 "'navbarIcon' => true," on Foreground Skin ] - phabricator.miraheze.org [16:17:04] Reception123: mobile and planning on enjoying the start of the new year [16:17:49] RhinosF1: ok [16:18:27] If I haven’t forgot by later in the week Reception123, yes [16:18:34] But i have poor memory [16:23:50] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown) [16:24:44] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown) [16:24:44] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [16:24:46] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown) [16:24:51] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 3 minutes ago with 8 failures. Failed resources (up to 3 shown) [16:24:52] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [16:25:00] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [16:25:08] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+1/-0/±0] 13https://git.io/JeA9t [16:25:09] [02miraheze/ssl] 07Reception123 0355e259f - add wiki.mnzp.xyz cert [16:25:10] [02ssl] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vxP9L [16:25:12] [02ssl] 07Reception123 opened pull request 03#251: add wiki.mnzp.xyz cert - 13https://git.io/JeA9q [16:25:13] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 428 failures. Last run 3 minutes ago with 428 failures. Failed resources (up to 3 shown) [16:25:14] paladox: ^ [16:25:20] Puppet gone mad? [16:25:21] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown) [16:25:27] puppet-server seems to be OOM'g [16:25:29] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 423 failures. Last run 3 minutes ago with 423 failures. Failed resources (up to 3 shown) [16:25:41] paladox: :( [16:25:42] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 424 failures. Last run 3 minutes ago with 424 failures. Failed resources (up to 3 shown) [16:25:44] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 423 failures. Last run 3 minutes ago with 423 failures. Failed resources (up to 3 shown): File[/etc/mathoid/config.yaml],File[/etc/apt/trusted.gpg.d/php.gpg],File[/usr/lib/php/20180731/luasandbox.so],File[/usr/lib/php/20170718/luasandbox.so] [16:25:45] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 48 failures. Last run 4 minutes ago with 48 failures. Failed resources (up to 3 shown) [16:25:47] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 413 failures. Last run 4 minutes ago with 413 failures. Failed resources (up to 3 shown) [16:25:56] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JeA9O [16:25:57] [02miraheze/ssl] 07Reception123 03127dbe7 - Update certs.yaml [16:25:58] Can you reboot it or clear the memory? [16:25:59] [02ssl] 07Reception123 synchronize pull request 03#251: add wiki.mnzp.xyz cert - 13https://git.io/JeA9q [16:25:59] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 426 failures. Last run 3 minutes ago with 426 failures. Failed resources (up to 3 shown): File[/usr/local/bin/gen_fingerprints],File[/etc/vim/vimrc.local],File[/usr/share/nginx/favicons/default.ico],File[/usr/share/nginx/favicons/apple-touch-icon-default.png] [16:26:21] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 129 failures. Last run 2 minutes ago with 129 failures. Failed resources (up to 3 shown): File[/etc/default/stunnel4],File[/etc/stunnel/mediawiki.conf],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate] [16:26:31] [02ssl] 07Reception123 closed pull request 03#251: add wiki.mnzp.xyz cert - 13https://git.io/JeA9q [16:26:32] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeA9s [16:26:34] [02miraheze/ssl] 07Reception123 0375edd09 - add wiki.mnzp.xyz cert (#251) * add wiki.mnzp.xyz cert * Update certs.yaml [16:26:35] [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-2 [16:26:36] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 190 failures. Last run 3 minutes ago with 190 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_nginx_errorrate],File[/etc/logrotate.d/salt-common],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [16:26:37] [02ssl] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vxP9L [16:26:44] [02ssl] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vxP9L [16:26:45] [02ssl] 07Reception123 deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:26:47] [02miraheze/ssl] 07Reception123 deleted branch 03paladox-patch-1 [16:26:48] [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-1 [16:27:01] Reception123: how you going to deploy until puppet wakes up? [16:27:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeA9Z [16:27:04] [02miraheze/mw-config] 07paladox 031381940 - Blcklist staff group Used by SocialProfile. Also blacklist editothersprofiles-private right. [16:27:53] paladox: same Q, shouldn’t we also be recovering puppet? [16:28:19] it'll recover in a few mins [16:28:23] miraheze/mw-config/master/1381940 - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/631339914 [16:28:52] paladox: don't deploy that [16:29:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeA9C [16:29:01] [02miraheze/mw-config] 07paladox 037679bdf - Fix syntax [16:29:16] SPF|Cloud i'm not :) [16:29:17] anyways fixed ^ [16:29:17] thanks [16:29:30] SPF|Cloud: see PM [16:29:37] PR sent is waiting on u [16:30:07] + the good news from Brent [16:30:21] miraheze/mw-config/master/7679bdf - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/631340448 [16:30:25] [02puppet] 07Southparkfan closed pull request 03#1172: Update donate@ - 13https://git.io/JeAmk [16:30:27] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeA9l [16:30:28] [02miraheze/puppet] 07RhinosF1 03e1a2497 - Update donate@ (#1172) * Update Now handled by Owen. Will contact Owen & Labster to notify * Rebase * Fix this rebase [16:32:36] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:32:37] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:32:46] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:33:00] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:33:11] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:33:16] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [16:33:16] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:33:28] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:33:46] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:33:55] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [16:33:59] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:34:21] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:34:32] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:34:34] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeA9z [16:35:09]