[02:56:43] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: A4A905B794A455AD2AF02C5D96040CA0BF11C87C from Apt::Source gluster_apt] [03:05:06] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:05:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jesu1 [03:05:10] [02miraheze/services] 07MirahezeSSLBot 0306aa1ab - BOT: Updating services config for wikis [06:27:18] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3023 MB (12% inode=94%); [09:06:46] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: A4A905B794A455AD2AF02C5D96040CA0BF11C87C from Apt::Source gluster_apt] [09:16:45] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [09:37:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [09:38:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [09:39:49] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:40:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [09:40:45] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [09:42:21] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [09:43:15] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 0.640 second response time [09:47:43] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 57% [09:48:08] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [09:48:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:48:49] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [09:49:09] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [09:50:37] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 5% [09:50:39] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [09:58:55] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [10:00:20] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [10:03:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:08:18] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [10:09:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [10:12:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:13:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [10:13:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:16:13] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [11:18:51] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown) [11:19:18] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [11:37:19] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [11:38:08] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:39:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [11:39:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [11:40:26] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [11:41:10] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.390 second response time [11:46:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:56:23] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [11:58:32] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:05:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JesrA [12:05:10] [02miraheze/services] 07MirahezeSSLBot 0393d510a - BOT: Updating services config for wikis [12:06:43] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: A4A905B794A455AD2AF02C5D96040CA0BF11C87C from Apt::Source gluster_apt] [12:10:11] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [12:16:11] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:34:41] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:36:34] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [12:36:55] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:37:43] looking [12:37:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:37:53] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [12:37:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [12:41:09] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 8.840 second response time [12:45:24] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [12:48:42] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [12:49:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:49:13] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [12:49:14] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [12:49:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:43:16] Does anybody watch https://commons.miraheze.org/wiki/Miraheze_Commons:Community_noticeboard ? [14:43:18] [ Miraheze Commons:Community noticeboard - Miraheze Commons ] - commons.miraheze.org [14:43:39] No but I'll look [15:10:16] I try too [15:16:02] paladox: https://meta.miraheze.org/w/index.php?diff=83643&oldid=83635&rcid=296905 [15:16:03] [ Difference between revisions of "Meta:Administrators' noticeboard" - Miraheze Meta ] - meta.miraheze.org [15:16:29] thanks, but https://splatoonwiki.miraheze.org/ is not on the new mounts. [15:16:31] [ Splatoon 中文維基 ] - splatoonwiki.miraheze.org [15:16:45] meta is though [15:17:03] paladox: hmm, that doesn't make sense then unless it's been an earlier issue we didn't spot [15:17:28] An issue? On miraheze? Lies [15:17:38] if that wiki is 503, it's because loginwiki 503'd [15:17:57] If loginwiki 503’s that means the world is over [15:17:57] paladox: ah k [15:59:51] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2615 MB (10% inode=94%); [16:06:26] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JesPR [16:06:27] [02miraheze/CreateWiki] 07translatewiki 03e8cdecf - Localisation updates from https://translatewiki.net. [16:06:28] [ Main page - translatewiki.net ] - translatewiki.net. [16:08:08] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.95, 1.58, 1.35 [16:10:59] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.43, 1.49, 1.35 [17:26:57] ZppixBot: i thought you were set to ignore Not-f817 [17:28:21] Zppix: it mustn't like you today [17:28:53] RhinosF1: It may not be idk, and i dont have access to look at its config rn, and if thats it, then add it to the list [17:30:44] Zppix: maybe something changed [17:34:36] RhinosF1: the only people who can change that config is me, Void, Reception123, or basically anyone in .botadmins [17:34:50] you have to have ssh access [17:35:04] Zppix: with the host of Not* or something [17:35:29] It hasn't restarted it something and sneakily changed nick has it? [17:36:05] Hmm it hasn't [17:36:22] RhinosF1: oh i was imagining it, i have an older config on a private paste and it has no ignores [17:36:37] RhinosF1: i guess it just never found a valid url besides git.io in Not-f817 [17:36:38] Zppix: oh [18:48:39] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [18:55:21] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:43:05] hi [19:44:06] hi [19:45:12] Hi [20:00:26] :) [20:02:02] How r u? [20:02:59] perfect! RhinosF1 [20:03:15] Good to hear! [20:04:10] hope you too. RhinosF1 [20:04:27] I'm good, a bit tired [20:04:42] ok :) [20:21:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jesy2 [20:22:00] [02miraheze/mw-config] 07paladox 0365cee2c - Move fapceowiki to use the new mount [20:29:00] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [20:29:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jesy6 [20:29:07] [02miraheze/puppet] 07paladox 034590452 - gluster: Fix apt key [20:36:34] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 4 minutes ago with 6 failures. Failed resources (up to 3 shown) [20:36:43] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown) [20:37:08] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 188 failures. Last run 4 minutes ago with 188 failures. Failed resources (up to 3 shown) [20:37:25] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown) [20:37:26] 188 failures, paladox what did you break [20:37:33] Zppix: daily GC [20:37:34] that's gc on puppet [20:37:46] thats some aggressive GC [20:38:12] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 151 failures. Last run 5 minutes ago with 151 failures. Failed resources (up to 3 shown) [20:38:14] Zppix: puppet restarts or something and it can have varying effects but it fails it [20:38:16] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=puppet1.miraheze.org&var-port=9100&fullscreen&panelId=78 [20:38:17] [ Grafana ] - grafana.miraheze.org [20:38:17] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 152 failures. Last run 5 minutes ago with 152 failures. Failed resources (up to 3 shown) [20:38:47] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 6 minutes ago with 9 failures. Failed resources (up to 3 shown) [20:39:14] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 7 minutes ago with 25 failures. Failed resources (up to 3 shown) [20:39:39] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 167 failures. Last run 6 minutes ago with 167 failures. Failed resources (up to 3 shown): File[reviwiki.info],File[reviwiki.info_private],File[wiki.tulpa.info],File[wiki.tulpa.info_private] [20:39:55] Zppix: if puppet is in a mood then giving it 10-15 mins in the naughty corner normally allows it to recover [20:40:01] lol [20:40:22] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 176 failures. Last run 7 minutes ago with 176 failures. Failed resources (up to 3 shown): File[monarchists.wiki],File[monarchists.wiki_private],File[stablestate.org],File[stablestate.org_private] [20:40:35] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 248 failures. Last run 7 minutes ago with 248 failures. Failed resources (up to 3 shown): File[wiki.kourouklides.com_private],File[wiki.ldmsys.net],File[wiki.ldmsys.net_private],File[wiki.ameciclo.org] [20:43:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jesyh [20:43:04] [02miraheze/mw-config] 07paladox 031042312 - Move breedersofthenephelymwiki to new mount [20:44:02] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:44:26] yey, Zppix, here comes the recovery spam [20:44:39] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:44:40] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:45:09] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:45:27] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:45:53] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:46:03] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:46:04] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:46:29] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:46:29] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:46:35] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:09:04] hey guys I'm back [21:09:09] been busy [21:09:10] hi [21:10:35] Examknow: they might still be a wiki request to do if you're free to [21:10:56] ok [21:11:24] sorry I havent been on lately but I have been busy with work stuff [21:11:29] np [21:26:07] RhinosF1: did my first request [21:27:03] Examknow: I'm assuming you didn't accept it or is feed not pinging me [21:27:37] I did not accept it [21:27:45] Examknow: k [21:27:45] the wiki name already existed [21:27:48] ah [21:28:41] Examknow: at least that error works properly now [21:28:57] yeah I tried to create it [21:29:03] it stoped me [21:29:32] Examknow: it used to fatal, then at one point it would say wiki exists instead of fataling at all [21:30:03] ah [21:32:25] paladox: 503 stunnel mw1/2 cp4 [21:32:42] ok [21:33:08] seems recovered? [21:33:42] paladox: ish, it's slow and icinga still reporting issues on datacentres now [21:34:16] hmm [21:37:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:39:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:39:45] paladox: ^ [21:39:56] yup, aware [21:41:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:41:10] yey [21:41:47] works for me [21:42:20] Examknow: 22:41:05 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online <--- it's back [21:42:33] and it depends on what server serves you [21:44:24] I know I was just saying that it was good [21:44:44] k [21:44:56] 3 datacenters are still down [21:45:20] Examknow: we're looking into it [21:48:09] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:48:22] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:48:39] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:49:04] ^known - ask paladox as i'm off to sleep [21:49:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [21:49:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:51:36] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:53:53] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:54:22] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:54:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:57:40] whats going on paladox ? [21:57:50] datacenter issues [21:57:52] Gluster causing slowdowns. [21:58:05] paladox: isnt that what gluster was meant to stop? [21:58:17] this is because gluster does not have a meta data service so isen't caching things in the ram like lizardfs [21:58:31] oh [21:58:38] well it caches a bit [21:58:43] but lizardfs is fully in the ram [21:58:46] so is performant [22:01:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [22:01:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [22:02:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:02:18] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [22:02:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:04:15] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:04:50] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:05:01] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:05:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:05:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:10:22] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:02:45] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:44] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2479 MB (10% inode=94%); [23:20:23] paladox: #diskspace #nonexistant