[00:12:07] <icinga-miraheze>	 PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 156.36 ms
[00:35:12] <icinga-miraheze>	 RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.63 ms
[01:45:07] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvD7l
[01:45:09] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot 03b7a9a08 - BOT: Updating services config for wikis
[02:11:44] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:12:13] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb
[02:15:19] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 1.198 second response time
[02:15:34] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[02:39:25] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::17f6/cpweb
[02:47:09] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[03:05:39] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[03:09:14] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[03:42:17] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[03:45:29] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[04:00:29] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2645 MB (10% inode=93%);
[04:58:00] <Not-a789>	 [02mw-config] 07soratako commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvDFn
[06:19:38] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 59 failures. Last run 5 minutes ago with 59 failures. Failed resources (up to 3 shown)
[06:19:46] <icinga-miraheze>	 PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 6 minutes ago with 10 failures. Failed resources (up to 3 shown)
[06:23:27] <icinga-miraheze>	 RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[06:24:24] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 43 failures. Last run 8 minutes ago with 43 failures. Failed resources (up to 3 shown): File[wiki.rmbrk.sk_private],File[vsfan.net],File[vsfan.net_private],File[infectowiki.com]
[06:26:52] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[06:27:33] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3565 MB (14% inode=93%);
[06:27:54] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:43:40] <RhinosF1>	 Reception123: around?
[11:01:42] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 6 minutes ago with 27 failures. Failed resources (up to 3 shown)
[11:02:22] <icinga-miraheze>	 PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 7 minutes ago with 10 failures. Failed resources (up to 3 shown)
[11:03:55] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 25 failures. Last run 7 minutes ago with 25 failures. Failed resources (up to 3 shown): File[electowiki.org],File[electowiki.org_private],File[drag.lgbt],File[drag.lgbt_private]
[11:05:00] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:05:37] <icinga-miraheze>	 RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[11:07:31] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:20:17] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:21:22] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:22:46] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:23:21] <icinga-miraheze>	 PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1377 bytes in 0.046 second response time
[11:23:33] <RhinosF1>	 what a morning
[11:24:32] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:24:44] <MH-Discord>	 <RhinosF1> @paladox: redis?
[11:25:29] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[11:27:28] <paladox>	 RhinosF1 i don't have time to look, also why did you ping me on discord when you wrote on irc? Please don't ping me in both. Secondly i think so by the sounds of things.
[11:27:28] <icinga-miraheze>	 PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:28:12] <RhinosF1>	 paladox: okay and MH-Discord did the ping on irc. I didn't even think about both.
[11:28:23] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw1 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:28:23] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[11:28:35] <icinga-miraheze>	 RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 18713 bytes in 2.971 second response time
[11:28:45] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw1 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:29:27] <icinga-miraheze>	 PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 3 backends are down. mw1 mw4 mw6
[11:29:47] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw1 mw6
[11:30:00] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:30:02] <icinga-miraheze>	 PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw1
[11:30:22] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw3 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:30:48] <icinga-miraheze>	 PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:31:01] <paladox>	 !log restart redis on rdb1
[11:31:36] <paladox>	 !log reboot rdb1
[11:31:37] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw3 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:32:08] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw1 mw2 mw3 mw4
[11:32:16] <icinga-miraheze>	 PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4225 bytes in 0.034 second response time
[11:33:24] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.751 second response time
[11:33:32] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw1 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15356 bytes in 3.488 second response time
[11:33:44] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw1 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15356 bytes in 5.214 second response time
[11:34:58] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18688 bytes in 1.411 second response time
[11:35:16] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.480 second response time
[11:35:30] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw1 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:35:39] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw3 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.053 second response time
[11:36:16] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw3 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 3.215 second response time
[11:37:00] <icinga-miraheze>	 RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 18687 bytes in 9.811 second response time
[11:37:10] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:40:12] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:40:13] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[11:40:28] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy
[11:40:40] <icinga-miraheze>	 RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy
[11:40:44] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw1 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15342 bytes in 0.040 second response time
[11:40:49] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.764 second response time
[11:40:55] <icinga-miraheze>	 RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18686 bytes in 0.548 second response time
[11:41:17] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[11:42:01] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 3.859 second response time
[11:42:31] <icinga-miraheze>	 RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 18673 bytes in 1.351 second response time
[11:43:05] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[11:43:49] <icinga-miraheze>	 RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy
[11:44:05] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.761 second response time
[11:44:06] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.005 second response time
[11:44:31] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[11:45:16] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18687 bytes in 0.521 second response time
[14:05:00] <Not-a789>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvyYg
[14:05:01] <Not-a789>	 [02miraheze/ManageWiki] 07translatewiki 03fee5dd2 - Localisation updates from https://translatewiki.net.
[14:05:02] <ZppixBot>	 [ Main page - translatewiki.net ] - translatewiki.net.
[14:43:59] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:44:17] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2607:5300:205:200::17f6/cpweb
[14:44:28] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:44:49] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.77.107.210/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[14:44:50] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:47:44] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.126 second response time
[14:48:01] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[14:48:09] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.409 second response time
[14:48:27] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[14:48:27] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.198 second response time
[14:58:50] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb
[15:05:13] <hispano76>	 Can anyone access my wikis? I get the same message as yesterday. https://privado.miraheze.org/wiki/  https://hispano.miraheze.org/wiki/  https://ucronias.miraheze.org/wiki/ 
[15:05:15] <ZppixBot>	 [ Internal error - Privado ] - privado.miraheze.org
[15:05:15] <ZppixBot>	 [ Internal error - Hispano ] - hispano.miraheze.org
[15:05:16] <ZppixBot>	 [ Internal error - Ucronías ] - ucronias.miraheze.org
[15:06:35] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:17:54] <paladox>	 !log MariaDB [privadowiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
[15:18:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:18:19] <paladox>	 !log MariaDB [hispanowiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
[15:18:42] <paladox>	 !log MariaDB [ucroniaswiki]> INSERT INTO slot_roles (`role_id`, `role_name`) VALUES (1, 'main'); - db7
[15:18:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:19:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:19:22] <paladox>	 hispano76 fixed
[15:22:43] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2607:5300:205:200::17f6/cpweb
[15:24:36] <hispano76>	 thanks, thanks paladox  
[15:24:57] <paladox>	 !log disable logging on mw[46] as an experiment
[15:25:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:26:26] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[15:31:47] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures
[15:31:52] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures
[15:32:11] <paladox>	 !log that is on nginx
[15:32:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:40:38] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[15:43:09] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvyGY
[15:43:11] <Not-a789>	 [02miraheze/puppet] 07paladox 0361ac6a9 - nginx: Buffer access logs
[15:43:12] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS
[15:43:14] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1295: nginx: Buffer access logs - 13https://git.io/JvyGO
[15:43:39] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1295: nginx: Buffer access logs - 13https://git.io/JvyGO
[15:43:40] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvyG3
[15:43:42] <Not-a789>	 [02miraheze/puppet] 07paladox 0313a5efb - nginx: Buffer access logs (#1295)
[15:43:43] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11
[15:43:45] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS
[15:44:51] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:49:13] <RhinosF1>	 paladox: testwiki is down
[15:49:20] <RhinosF1>	 ERR_CONNECTION_REFUSE
[15:49:29] <paladox>	 works for me?
[15:49:55] <RhinosF1>	 https://usercontent.irccloud-cdn.com/file/WYX3I5eW/Screenshot%202020-03-23%20at%2015.49.47.png
[15:50:17] <RhinosF1>	 weird back
[15:50:37] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:50:43] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:51:24] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 5 minutes ago with 35 failures. Failed resources (up to 3 shown): File[wiki.twilightsignal.com_private],File[wiki.fourta.org],File[wiki.fourta.org_private],File[wiki.alirezaivaz.ir]
[15:54:56] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[15:55:47] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[15:55:56] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvyGr
[15:55:58] <Not-a789>	 [02miraheze/puppet] 07paladox 03c655090 - Update nginx.conf.erb
[15:59:15] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 9.413 second response time
[15:59:20] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb
[16:00:32] <Not-a789>	 [02RottenLinks] 07Seb35 opened pull request 03#17: Fix some issues in 1.0.8 - 13https://git.io/JvyGy
[16:03:27] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:55:00] <Not-a789>	 [02RottenLinks] 07JohnFLewis closed pull request 03#17: Fix some issues in 1.0.8 - 13https://git.io/JvyGy
[16:55:01] <Not-a789>	 [02miraheze/RottenLinks] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±4] 13https://git.io/JvynV
[16:55:03] <Not-a789>	 [02miraheze/RottenLinks] 07Seb35 0397e417d - Fix some issues in 1.0.8
[16:55:04] <Not-a789>	 [02miraheze/RottenLinks] 07JohnFLewis 0343583d6 - Merge pull request #17 from WikiValley/T5345 Fix some issues in 1.0.8
[16:57:39] <Not-a789>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jvyni
[16:57:40] <Not-a789>	 [02miraheze/mediawiki] 07JohnFLewis 03e66747e - update RL
[17:16:35] <Not-a789>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvycG
[17:16:36] <Not-a789>	 [02miraheze/ssl] 07paladox 03da6d9c3 - Add madzebrascience.wiki ssl certificate
[17:16:38] <Not-a789>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[17:16:39] <Not-a789>	 [02ssl] 07paladox opened pull request 03#276: Add madzebrascience.wiki ssl certificate - 13https://git.io/JvycZ
[17:20:20] <Not-a789>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvyc2
[17:20:22] <Not-a789>	 [02miraheze/ssl] 07paladox 0301ec419 - Update madzebrascience.wiki.crt
[18:04:52] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[18:08:13] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.771 second response time
[18:15:59] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[18:16:04] <UrielMachine>	 hey there
[18:16:11] <Zppix>	   hi
[18:16:36] <Zppix>	 how are you UrielMachine ?
[18:16:40] <UrielMachine>	 I have two dilemmae: 1) i can't see much of the twitter feed. I just see some white bars. Have i been muted or blocked or something?
[18:16:55] <Zppix>	 UrielMachine:  white bars on what?
[18:17:07] <UrielMachine>	 The miraheze witter
[18:17:15] <UrielMachine>	 I can get a screenshot
[18:17:28] <Zppix>	 UrielMachine:  maybe just your browser or connection.
[18:17:52] <UrielMachine>	 Lemme check something: has the twitter been updated since march 8
[18:17:55] <Zppix>	 UrielMachine:  we have nothing new on twitter regardless
[18:18:07] <UrielMachine>	 Then, it's prolly on my end
[18:18:54] <Zppix>	 possibly
[18:19:48] <UrielMachine>	 anyhow, 2) My wiki got classified as inactive, even though i have been editing it almost daily. I had to mark it as private to save it. My only guess was that i was the sole user of the site. 
[18:19:50] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 5.076 second response time
[18:20:03] <UrielMachine>	 what is icinga talking about
[18:20:41] <Zppix>	 UrielMachine:  don't worry about icinga :)
[18:20:53] <Zppix>	 UrielMachine: I can unmark it as inactive for you
[18:20:56] <Zppix>	 UrielMachine:  which wiki is it
[18:21:03] <UrielMachine>	 Fireberg Media Database
[18:21:14] <Zppix>	 url?
[18:21:14] <UrielMachine>	 Does it mark wikis as inactive if only one person uses it?
[18:21:24] <Zppix>	 UrielMachine:  i think its based on activity
[18:21:28] <UrielMachine>	 https://fireberg.miraheze.org/wiki/Main_Page
[18:21:31] <ZppixBot>	 [ Fireberg Media Database ] - fireberg.miraheze.org
[18:21:48] <UrielMachine>	 I've been using it each day, or at least every other day.
[18:22:15] <Zppix>	 UrielMachine:  it doesnt appear to be marked as inactive
[18:22:33] <UrielMachine>	 Well, i unmarked it as such recently, i think. I made the server private. 
[18:22:56] <UrielMachine>	 But now, i have two different questions: 1) are there any stipulations on activity for public wikis?
[18:23:19] <UrielMachine>	 and 2) Are the databases slow today? Trying to see if the problem is on your end too.
[18:23:28] <Zppix>	 UrielMachine:  our dormacy policy is outlined here: https://meta.miraheze.org/wiki/Dormancy_Policy
[18:23:30] <ZppixBot>	 [ Dormancy Policy - Miraheze Meta ] - meta.miraheze.org
[18:23:34] <UrielMachine>	 grax
[18:23:52] <Zppix>	 UrielMachine:  it can be slow at times
[18:24:13] <UrielMachine>	 2x graxx
[18:24:22] <UrielMachine>	 Thank you so much for the help Zppix
[18:24:55] <Zppix>	 np
[18:25:40] <UrielMachine>	 Feel free to visit Fireberg sometime. I'm trying to make it an IMDb for any type of role that can't be confirmed with the eyes (voice acting, motion capture, costumes, literally anything behind the scenes, music)
[18:26:00] <UrielMachine>	 But in practice, it's just a hub for me making linguistic and conlinguistic journals
[18:26:25] <UrielMachine>	 peace
[18:28:57] <MikeV>	 VisualEditor (parsoid?) failing again on nenawiki and ndgnenawiki: "Error loading data from server: apierror-visualeditor-docserver-http: HTTP 503. Would you like to retry?"
[18:29:20] <Reception123>	 paladox: ^
[18:29:54] <paladox>	 503 indicates server is down
[18:33:13] <MikeV>	 paladox: I can edit source.
[18:33:49] <paladox>	 Works now
[18:34:12] <paladox>	 !log service parsoid stop ; killall -u parsoid ; service parsoid start - services[12]
[18:34:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:34:47] <MikeV>	 Somebody chain paladox to his workstation.  Then wash your hands.
[18:35:09] <paladox>	 heh
[18:43:24] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[18:47:35] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.005 second response time
[18:48:29] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb
[18:52:42] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[19:14:39] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.77.107.210/cpweb, 2607:5300:205:200::17f6/cpweb
[19:18:51] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[19:44:06] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb
[19:50:44] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:52:48] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[19:54:37] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.006 second response time
[20:12:00] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/opendkim/KeyTable]
[20:16:12] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[20:23:43] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:27:32] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time
[20:28:01] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:28:03] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:28:49] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb
[20:35:08] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:36:57] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[20:38:59] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.488 second response time
[20:39:24] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.823 second response time
[20:39:27] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.976 second response time
[20:58:16] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[21:02:37] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[21:05:20] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb
[21:08:59] <Not-a789>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvyBE
[21:09:00] <Not-a789>	 [02miraheze/ManageWiki] 07JohnFLewis 03b33759e - minor changes to ManageWiki with display and processing
[21:09:02] <JohnLewis>	 paladox: ^ look good?
[21:09:07] * paladox looks
[21:09:17] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:09:44] <paladox>	 JohnLewis couldn't we introduce a config for https://github.com/miraheze/ManageWiki/compare/fee5dd2af943...b33759eed901#diff-32e0691b16e504105802b793f05fac35R1304 ?
[21:09:44] <ZppixBot>	 [ Comparing fee5dd2af943...b33759eed901 · miraheze/ManageWiki · GitHub ] - github.com
[21:10:06] <JohnLewis>	 for what generalised purpose?
[21:10:29] <paladox>	 To make it configurable? Maybe other farms have groups that they don't want users to delete?
[21:10:39] <paladox>	 Other then that, your change looks perfect to me!
[21:13:52] <Not-a789>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvyBr
[21:13:54] <Not-a789>	 [02miraheze/ManageWiki] 07JohnFLewis 0383ccb2c - adjust configurability for permanent groups
[21:13:55] <JohnLewis>	 paladox: that?
[21:14:20] <paladox>	 yup! Thanks JohnLewis!
[21:14:59] <Not-a789>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvyBK
[21:15:00] <Not-a789>	 [02miraheze/mediawiki] 07JohnFLewis 033397b73 - update MW
[21:36:51] <ZppixBot>	 Hello wm-bot574! If you have any questions, feel free to ask and someone should answer soon.
[21:52:42] <icinga-miraheze>	 PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[22:04:46] <icinga-miraheze>	 RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:08:26] <RhinosF1>	 Saw the welcome message, that’s a wm-bot574 bug not a ZppixBot over welcoming bug
[23:40:05] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[23:48:29] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:50:21] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 1 failures
[23:58:23] <icinga-miraheze>	 PROBLEM - test2 HTTPS on test2 is CRITICAL: connect to address 51.77.107.211 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket