[00:00:41] RECOVERY - gluster1 Disk Space on gluster1 is OK: DISK OK - free space: / 75771 MB (11% inode=83%); [00:16:54] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:33] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb [00:18:45] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:55] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:26] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.317 second response time [00:22:10] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:39] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 4.885 second response time [00:22:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:23:37] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:07] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.721 second response time [00:27:30] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18674 bytes in 0.759 second response time [00:28:56] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.004 second response time [00:34:33] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.251 second response time [01:06:55] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2643 MB (10% inode=93%); [01:18:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::17f6/cpweb [01:22:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:35:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [01:38:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:00:15] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:00:25] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:04:14] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.855 second response time [02:04:18] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.312 second response time [02:14:43] so if i message here, it morphs into irc? [02:15:16] awesome [02:34:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [02:42:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:45:19] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:45:19] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:45:26] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:45:30] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:46:29] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:46:30] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:46:30] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:46:46] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:15] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:15] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:28] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:28] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:37] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:41] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:41] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 7 minutes ago with 17 failures. Failed resources (up to 3 shown) [02:47:42] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:42] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:42] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:46] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:46] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:47] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:47:52] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:48:02] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:48:02] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:48:04] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:48:55] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:48:57] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:49:13] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:49:15] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:52:42] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:52:43] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [02:52:43] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:53:34] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:53:35] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:53:35] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:53:47] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:54:38] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:54:42] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:08] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:09] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:18] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:55:19] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:23] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:27] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:55:27] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:27] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:31] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:32] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:55:32] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:55:37] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:46] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:55:46] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:55:48] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:56:31] PROBLEM - gluster1 Disk Space on gluster1 is WARNING: DISK WARNING - free space: / 75616 MB (10% inode=83%); [02:56:38] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:56:49] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:56:53] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:56:58] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:56:58] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:48:58] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:52:47] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 5.077 second response time [04:08:47] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:11:38] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:12:48] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.701 second response time [04:15:20] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.315 second response time [04:18:27] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.62, 1.80, 1.70 [04:22:19] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.40, 0.98, 1.38 [05:33:13] PROBLEM - ping4 on misc2 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:33:21] PROBLEM - misc2 Current Load on misc2 is CRITICAL: connect to address 81.4.127.174 port 5666: No route to hostconnect to host 81.4.127.174 port 5666: No route to host [05:33:30] PROBLEM - misc2 Disk Space on misc2 is CRITICAL: connect to address 81.4.127.174 port 5666: No route to hostconnect to host 81.4.127.174 port 5666: No route to host [05:34:49] PROBLEM - test1 SSH on test1 is CRITICAL: connect to address 185.52.2.243 and port 22: No route to host [05:35:28] PROBLEM - Host test1 is DOWN: PING CRITICAL - Packet loss = 100% [05:35:48] PROBLEM - misc2 Redis Process on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:35:53] PROBLEM - misc2 SSH on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:14] PROBLEM - Host misc2 is DOWN: PING CRITICAL - Packet loss = 100% [06:27:54] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3437 MB (14% inode=93%); [07:36:25] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:36:35] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:41:52] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 4.460 second response time [07:42:06] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 3.317 second response time [07:42:55] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 280 failures. Last run 9 minutes ago with 280 failures. Failed resources (up to 3 shown): File[wiki.warfra.ml_private],File[prfm.wiki],File[prfm.wiki_private],File[encyclopediaofastrobiology.org] [07:43:20] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 285 failures. Last run 10 minutes ago with 285 failures. Failed resources (up to 3 shown): File[wiki.doverdistrictscouts.co.uk],File[wiki.doverdistrictscouts.co.uk_private],File[wiki.veloren.net],File[wiki.veloren.net_private] [07:43:53] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 147 failures. Last run 9 minutes ago with 147 failures. Failed resources (up to 3 shown): File[pwiki.arkcls.com],File[pwiki.arkcls.com_private],File[kunwok.org],File[kunwok.org_private] [07:45:18] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 45 failures. Last run 9 minutes ago with 45 failures. Failed resources (up to 3 shown): File[thelonsdalebattalion.co.uk],File[thelonsdalebattalion.co.uk_private],File[allthetropes.org],File[allthetropes.org_private] [07:48:15] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:48:39] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [07:48:59] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [07:50:20] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [10:58:43] RECOVERY - Host test1 is UP: PING OK - Packet loss = 44%, RTA = 13.36 ms [10:58:46] PROBLEM - test1 Disk Space on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:58:47] PROBLEM - test1 php-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:58:47] PROBLEM - ping4 on test1 is CRITICAL: PING CRITICAL - Packet loss = 100% [10:58:47] PROBLEM - test1 Current Load on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refusedconnect to host 185.52.2.243 port 5666: Connection refused [10:58:47] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:41] PROBLEM - test1 php-fpm on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refusedconnect to host 185.52.2.243 port 5666: Connection refused [11:00:01] PROBLEM - ping4 on test1 is CRITICAL: PING CRITICAL - Packet loss = 100% [11:00:03] RECOVERY - ping4 on test1 is OK: PING OK - Packet loss = 0%, RTA = 13.11 ms [11:00:36] PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [11:00:41] PROBLEM - test1 Disk Space on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refusedconnect to host 185.52.2.243 port 5666: Connection refused [11:01:01] PROBLEM - test1 Current Load on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refusedconnect to host 185.52.2.243 port 5666: Connection refused [11:01:56] RECOVERY - test1 SSH on test1 is OK: SSH OK - OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 (protocol 2.0) [11:27:58] PROBLEM - ping4 on test1 is WARNING: PING WARNING - DUPLICATES FOUND! Packet loss = 0%, RTA = 13.55 ms [11:28:28] Reception123: ^ ?????????????????????????? [11:37:08] no idea, ask paladox [11:40:38] Reception123: it's decom, icinga-miraheze needs to relax [11:50:10] .in 30mins check log_raw [11:50:12] RhinosF1: Okay, I will set the reminder for: 2020-03-25 - 12:20:12GMT [12:20:12] RhinosF1: check log_raw [12:33:21] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:37:26] PROBLEM - cp8 Stunnel Http for mw3 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:38:14] PROBLEM - cp7 Stunnel Http for mw3 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:38:49] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.702 second response time [12:39:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [12:40:12] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:41:40] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1377 bytes in 0.039 second response time [12:42:04] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 3 backends are down. mw2 mw4 mw6 [12:42:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 3 backends are down. mw2 mw4 mw6 [12:42:34] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw2 mw4 mw6 [12:42:55] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:02] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 2 backends are down. mw2 mw6 [12:43:29] RECOVERY - cp8 Stunnel Http for mw3 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.353 second response time [12:44:14] RECOVERY - cp7 Stunnel Http for mw3 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.053 second response time [12:45:52] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.486 second response time [12:46:44] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18686 bytes in 0.524 second response time [12:46:54] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [12:47:02] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [12:47:07] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [12:47:14] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 18668 bytes in 0.512 second response time [12:47:14] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [12:49:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:02:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [14:07:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:11:44] RhinosF1 why are you asking Reception123 if you know its decom'd? [14:12:10] test1 has been deleted [14:12:30] if it's deleted, why is icinga talking about it? [14:13:47] maybe someone forgot to delete it from the config? [14:15:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvS26 [14:15:10] [02miraheze/services] 07MirahezeSSLBot 03a4c8bf9 - BOT: Updating services config for wikis [14:35:57] JohnLewis: modern is a default skin! We might have an issue adding that as conflicting VE! [14:36:22] Because for some reason it hasn't been removed from puppetdb resources. [14:36:41] RhinosF1: yeah, kind of impossible that way [14:37:14] JohnLewis: we'll just hope no one else notices then! Only one user reported it. [15:05:25] Hi [15:06:17] hi [15:09:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [15:15:33] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:20:25] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 18669 bytes in 0.596 second response time [15:36:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [15:38:34] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/JvSV7 [15:38:36] [02miraheze/mediawiki] 07Reception123 03b5b7174 - Add SpriteSheet [15:38:41] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:09] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.275 second response time [15:50:06] Reception123: where are you from as I need someone who understands Dutch education? [15:52:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:53:32] RhinosF1: I definitely don't, SPF|Cloud can help with that [15:54:05] SPF|Cloud: you around because this could become a ToU issue [16:11:41] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:16:24] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [16:41:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [17:05:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:38:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [17:40:37] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:16] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:41:49] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:42:43] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:44] PROBLEM - cp7 Stunnel Http for mw3 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:44:34] PROBLEM - cp8 Stunnel Http for mw3 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:45:20] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [17:47:00] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.188 second response time [17:47:40] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:27] RECOVERY - cp7 Stunnel Http for mw3 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.056 second response time [17:48:48] RhinosF1: you can close the upstream task for the issue we discussed yesterday [17:48:53] I've found a JS work around [17:49:28] RECOVERY - cp8 Stunnel Http for mw3 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.611 second response time [17:50:22] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:32] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:51:48] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 6.043 second response time [17:52:40] .in 2hour close MW upstream for John [17:52:41] RhinosF1: Okay, I will set the reminder for: 2020-03-25 - 19:52:41GMT [17:52:52] JohnLewis: ^ [17:53:05] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18695 bytes in 0.533 second response time [17:58:15] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 3.098 second response time [17:58:50] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 6.877 second response time [17:59:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:00:32] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.711 second response time [18:01:31] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvS6m [18:01:33] [02miraheze/CreateWiki] 07JohnFLewis 039848234 - change js from onhashchange to onload [18:02:01] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 3.173 second response time [18:02:02] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [18:02:03] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [18:02:14] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvS6O [18:02:15] [02miraheze/mediawiki] 07JohnFLewis 0362f8bf4 - update CW [18:04:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:06:52] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [18:06:52] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [18:54:59] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:55:11] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:59:22] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.672 second response time [18:59:30] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.322 second response time [19:52:42] RhinosF1: close MW upstream for John [20:00:11] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:14] .in 2hours 19:52:42 RhinosF1: close MW upstream for John [20:02:15] RhinosF1: Okay, I will set the reminder for: 2020-03-25 - 22:21:14GMT [20:02:21] ZppixBot: let me procrastinate [20:04:52] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.078 second response time [20:09:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [20:14:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:15:59] paladox: security pre release, update tommorow [20:16:09] yup, already know [20:18:06] Kk [20:20:02] hi RhinosF1 paladox :) [20:20:35] hi [20:25:17] Hi [20:27:02] There's 7 PRs open on mw-config, can someone go through them please? paladox / Reception123 [20:27:17] sure [20:27:35] There's a lot of PRs open across the whole org tbf [20:27:49] [02mw-config] 07paladox closed pull request 03#2939: Switch db6 to db7 - 13https://git.io/Jvoe5 [20:27:52] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [20:27:53] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [20:28:09] [02mw-config] 07paladox closed pull request 03#2944: Add MIME extensions - 13https://git.io/JvPcE [20:28:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvS16 [20:28:12] [02miraheze/mw-config] 07cmgi201 03d327da1 - Add MIME extensions (#2944) There is a .bnk (for moodle bank question file), .mbz (for course moodle export format) and .scss (for theme scss moodle format. Thanks [20:28:25] [02mw-config] 07paladox closed pull request 03#2936: Set db4 into read only - 13https://git.io/JvrrN [20:28:28] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:28:29] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:29:46] [02mw-config] 07paladox closed pull request 03#2824: Adjust protection settings - 13https://git.io/JeH12 [20:29:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvS11 [20:29:49] [02miraheze/mw-config] 07PlavorSeol 03f89d5f0 - Adjust protection settings (#2824) * Adjust protection settings * Add empty string for unprotecting ("Allow all users") Forgot to add this * Update LocalSettings.php 그래 당신 원하는대로 해주마 [20:30:19] paladox: have you checked that? we don’t allow every file extension. [20:30:55] Yes, its fine to add the file extensions, since not like extensions will have it enabled just because we add it. [20:30:59] Its case by case [20:33:03] Ok [20:34:19] I’m not even sure https://github.com/miraheze/mw-config/commit/f89d5f09e877f8697898e5de39a7f55dc9b93d13 has any impact [20:34:20] [ Adjust protection settings (#2824) · miraheze/mw-config@f89d5f0 · GitHub ] - github.com [20:48:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.77.107.210/cpweb, 2607:5300:205:200::17f6/cpweb [20:49:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:1056::2/cpweb [20:53:10] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:44] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:54:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:54:47] [02mw-config] 07paladox commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvSMl [20:55:51] [02mw-config] 07paladox closed pull request 03#2945: Increase wgHTTPTimeout and wgHTTPConnectTimeout to 60 - 13https://git.io/JvXBF [20:55:53] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [20:55:55] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [20:56:44] [02miraheze/mw-config] 07paladox deleted branch 03Southparkfan-patch-1 [20:56:46] [02mw-config] 07paladox deleted branch 03Southparkfan-patch-1 - 13https://git.io/vbvb3 [20:56:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [20:56:57] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [20:56:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [20:57:00] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [20:57:13] [02puppet] 07paladox closed pull request 03#1229: mediawiki: Switch mw1 to jobrunner1 for letsencrypt and services - 13https://git.io/JvlEg [20:57:17] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [20:57:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [20:58:34] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.137 second response time [21:00:33] paladox: SRE need to discuss #2942 as a group. You removed v3 before. [21:01:47] SRE don't need to discuss it as a group [21:03:06] ^ [21:04:07] JohnLewis: why did we choose to drop v3 in the first place? [21:04:38] Also as paladox points out, anyone that’s edited the wiki will have to agree to re-license their contributions. [21:04:48] v4 was the latest and is an international license. v3 had to be ported to local jurisdictions [21:05:42] Hmm [21:05:50] That makes sense [21:06:51] Going v3 -> v4 is covered but v4 -> v3 isn’t directly compatible afaik [21:07:01] Owen should be able to confirm [21:07:54] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.761 second response time [21:14:52] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [21:15:30] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.002 second response time [21:15:36] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.762 second response time [21:19:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:20:19] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 3.195 second response time [21:20:24] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 3.385 second response time [21:20:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb [21:25:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:40:08] [02mw-config] 07paladox commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvSD9 [21:40:31] paladox, JohnLewis: can we make file uploads private on https://mikekilitterbox.miraheze.org/wiki per request on discord [21:40:57] [ Mikeki's Litterbox ] - mikekilitterbox.miraheze.org [21:42:55] [02ManageWiki] 07paladox closed pull request 03#133: modifyGroupPermission: Fix defaults for options - 13https://git.io/JvqP3 [21:42:57] [02ManageWiki] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vpSns [21:42:59] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-2 [21:43:48] [02DataDump] 07paladox commented on pull request 03#5: Polish i18n - 13https://git.io/JvSD5 [21:45:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [21:46:50] [02MatomoAnalytics] 07paladox synchronize pull request 03#10: Add composer.json - 13https://git.io/Je0TY [21:49:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:50:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb [21:53:46] [02MatomoAnalytics] 07paladox closed pull request 03#10: Add composer.json - 13https://git.io/Je0TY [21:53:48] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvSyq [21:53:49] [02miraheze/MatomoAnalytics] 07dgrammlich 036aab7dd - Add composer.json (#10) * Add composer.json * Update composer.json Co-authored-by: paladox [21:58:53] [02MatomoAnalytics] 07paladox closed pull request 03#17: Add Persian translation - 13https://git.io/Jvk5M [21:58:55] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvSyg [21:58:56] [02miraheze/MatomoAnalytics] 07AlirezaIvaz 03a618364 - Add Persian translation (#17) [21:59:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:00:30] paladox: did you see my message above? [22:02:41] yup, seen now. [22:03:11] [02MirahezeMagic] 07paladox closed pull request 03#101: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/Jvqza [22:03:12] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSy6 [22:03:14] [02miraheze/MirahezeMagic] 07paladox 036f3b499 - Replace deprecated isAllowed(...) with userHasRight(...) (#101) Bug: T5008 [22:03:15] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [22:03:17] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [22:05:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSyX [22:05:03] [02miraheze/mw-config] 07paladox 03cacf662 - mikekilitterboxwiki: Make files private [22:06:07] RhinosF1 done [22:07:21] Thx, seen [22:11:11] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvSyy [22:11:12] [02miraheze/IncidentReporting] 07paladox 03fba3304 - Replace deprecated isAllowed(...) with userHasRight(...) [22:11:14] [02IncidentReporting] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fh5YJ [22:11:21] [02IncidentReporting] 07paladox opened pull request 03#10: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/JvSyS [22:12:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:1056::2/cpweb [22:13:12] miraheze/IncidentReporting/paladox-patch-3/fba3304 - paladox The build passed. https://travis-ci.com/miraheze/IncidentReporting/builds/155238750 [22:13:52] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvSyQ [22:13:54] [02miraheze/IncidentReporting] 07paladox 03b8b2f75 - Update SpecialIncidentReports.php [22:13:55] [02IncidentReporting] 07paladox synchronize pull request 03#10: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/JvSyS [22:14:14] [02IncidentReporting] 07paladox synchronize pull request 03#10: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/JvSyS [22:14:16] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvSyF [22:14:17] [02miraheze/IncidentReporting] 07paladox 0377c49fb - Update IncidentReportingFormFactory.php [22:14:48] [02IncidentReporting] 07paladox synchronize pull request 03#10: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/JvSyS [22:14:50] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvSyb [22:14:51] [02miraheze/IncidentReporting] 07paladox 036e1adf2 - Update SpecialIncidentReports.php [22:15:28] [02IncidentReporting] 07paladox commented on pull request 03#9: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/JvSyA [22:15:29] [02IncidentReporting] 07paladox closed pull request 03#9: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/Jvqze [22:15:31] [02IncidentReporting] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fh5YJ [22:15:32] [02miraheze/IncidentReporting] 07paladox deleted branch 03paladox-patch-2 [22:21:02] [02IncidentReporting] 07paladox closed pull request 03#10: Replace deprecated isAllowed(...) with userHasRight(...) - 13https://git.io/JvSyS [22:21:04] [02miraheze/IncidentReporting] 07paladox pushed 035 commits to 03master [+0/-0/±6] 13https://git.io/JvSSU [22:21:05] [02miraheze/IncidentReporting] 07paladox 03e95a692 - Merge pull request #10 from miraheze/paladox-patch-3 Replace deprecated isAllowed(...) with userHasRight(...) [22:21:07] [02IncidentReporting] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fh5YJ [22:21:08] [02miraheze/IncidentReporting] 07paladox deleted branch 03paladox-patch-3 [22:21:14] RhinosF1! [22:21:30] ZppixBot: yes? [22:22:23] Oh I know why [22:24:04] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:24:12] JohnLewis: 22:23:48 User-RhinosF1, MediaWiki-HTMLForm, OOUI: FormFactory doesn’t show the default tab as selected on first view - https://phabricator.wikimedia.org/T248411 (RhinosF1) Open→Invalid We found a bug in our code and fixed this. [22:24:13] [ ⚓ T248411 FormFactory doesn’t show the default tab as selected on first view ] - phabricator.wikimedia.org [22:24:50] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:27:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:29:05] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.303 second response time [22:29:41] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.363 second response time [22:34:13] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvSSR [22:34:15] [02miraheze/ManageWiki] 07paladox 03e54a19d - ManageWiki: Add hook to fnNewSidebarItem [22:34:16] [02ManageWiki] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vpSns [22:34:18] [02ManageWiki] 07paladox opened pull request 03#136: ManageWiki: Add hook to fnNewSidebarItem - 13https://git.io/JvSS0 [22:34:33] [02ManageWiki] 07paladox synchronize pull request 03#136: ManageWiki: Add hook to fnNewSidebarItem - 13https://git.io/JvSS0 [22:34:34] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvSSE [22:34:36] [02miraheze/ManageWiki] 07paladox 033b309f9 - Update ManageWikiHooks.php [22:38:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [22:46:40] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvSSi [22:46:42] [02miraheze/DataDump] 07paladox 036be6f40 - Use SkinBuildSidebar hook [22:46:43] [02DataDump] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fhhKV [22:46:45] [02DataDump] 07paladox opened pull request 03#6: Use SkinBuildSidebar hook - 13https://git.io/JvSSP [22:52:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:04:17] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvS9s [23:04:18] [02miraheze/DataDump] 07paladox 03d898d71 - Update DataDumpHooks.php [23:04:20] [02DataDump] 07paladox synchronize pull request 03#6: Use SkinBuildSidebar hook - 13https://git.io/JvSSP [23:07:53] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:13] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:45] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 4.039 second response time [23:12:59] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 6.574 second response time [23:24:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvS9H [23:24:32] [02miraheze/mw-config] 07paladox 0350286a5 - Set timeout to 60s by default [23:24:33] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:24:35] [02mw-config] 07paladox opened pull request 03#2948: Set timeout to 60s by default - 13https://git.io/JvS9Q [23:24:54] [02mw-config] 07paladox synchronize pull request 03#2948: Set timeout to 60s by default - 13https://git.io/JvS9Q [23:24:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvS97 [23:24:57] [02miraheze/mw-config] 07paladox 035829bdd - Update LocalSettings.php [23:26:04] [02mw-config] 07paladox synchronize pull request 03#2948: Set timeout to 60s by default - 13https://git.io/JvS9Q [23:26:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvS95 [23:26:07] [02miraheze/mw-config] 07paladox 0319518b9 - Update LocalSettings.php [23:26:20]