[00:27:14] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-51.89.160.136-to-any-port-443] [00:33:21] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:42:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.161.32.127/cpweb [00:49:37] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw1 [00:53:46] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [00:57:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:01:18] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:02:05] PROBLEM - services1 Puppet on services1 is WARNING: WARNING: Puppet last ran 1 hour ago [01:04:23] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:05:04] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 3.516 second response time [01:05:34] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:07:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [01:07:55] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [01:10:10] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:25:31] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2925 MB (12% inode=94%); [06:50:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPxe [06:50:09] [02miraheze/services] 07MirahezeSSLBot 03a6a947a - BOT: Updating services config for wikis [14:46:01] !log rebuildrc and initsitestats on libertygamewiki following successful import [14:46:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:28:23] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:28:33] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:29:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [18:33:36] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:10] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:34:33] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:35:19] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw5 [18:35:20] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [18:35:34] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw5 [18:35:35] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw5 [18:36:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:37:12] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.006 second response time [18:37:31] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.309 second response time [18:37:42] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [18:38:24] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [18:38:27] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [18:38:41] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [18:38:41] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [18:39:12] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.006 second response time [18:39:13] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [19:22:38] Hello cranberries! If you have any questions, feel free to ask and someone should answer soon. [19:22:40] I'm sure you guys missed me ! For St. Patricks day it's time to sing along !! I hope you all are in the mood of singing to cheer up the sadness because of the coronavirus [19:22:43] I'm sure you guys missed me ! For St. Patricks day it's time to sing along !! I hope you all are in the mood of singing to cheer up the sadness because of the coronavirus [19:22:47] Another head hangs lowly [19:22:51] Child is slowly taken [19:22:56] And the violence, caused such silence [19:23:01] Who are we mistaken? [19:23:05] In your head, in your head [19:23:10] Zombie, zombie, zombie-ie-ie [19:23:14] What's in your head, in your head [19:23:18] Zombie, zombie, zombie-ie-ie, oh [19:23:23] Du, du, du, du [19:23:26] Du, du, du, du [19:23:31] Du, du, du, du [19:23:36] Another mother's breaking [19:23:42] Heart is taking over [19:23:47] When the violence causes silence [19:23:53] We must be mistaken [19:23:58] It's the same old theme [19:24:03] Since nineteen-sixteen [19:24:09] In your head, in your head, they're still fighting [19:24:12] With their tanks, and their bombs [19:24:17] And their bombs, and their guns [19:24:23] In your head, in your head, they are dying [19:24:29] In your head, in your head [19:24:34] Zombie, zombie, zombie-ie-ie [19:24:38] What's in your head, in your head [19:24:43] Zombie, zombie, zombie-ie-ie, oh oh oh oh oh oh oh ie-ie oh [19:25:08] Come on don't be shy staff sing along with me ! it's fun !! Zppix Voidwalker Reception123 RhinosF1 [19:25:10] Come on don't be shy staff sing along with me ! it's fun !! Zppix Voidwalker Reception123 RhinosF1 [19:25:11] * cranberries Come on don't be shy staff sing along with me ! it's fun !! Zppix Voidwalker Reception123 RhinosF1 [19:25:12] Come on don't be shy staff sing along with me ! it's fun !! Zppix Voidwalker Reception123 RhinosF1 [19:35:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvX8b [19:35:09] [02miraheze/services] 07MirahezeSSLBot 0354bd1c3 - BOT: Updating services config for wikis [19:41:59] Hello whitestripes[m]! If you have any questions, feel free to ask and someone should answer soon. [19:42:10] PartyRhino That's the spirit it's party time for St. Patrick's day now SING with me !! [19:44:39] Hello whitestripes2[m]! If you have any questions, feel free to ask and someone should answer soon. [19:45:30] need some +o there paladox? [19:45:41] Apparently it was slow :P [20:35:38] We're getting "Something went wrong" HTTP 500 error when trying to save. [20:36:20] paladox: ^ [20:43:28] MikeV which wiki? And do you still have that issue? [20:45:45] neg.nenawiki.org [20:45:56] ndg.nenawiki.org [20:47:51] It just failed again. Now: Unknown error, HTTP status 0 [20:48:03] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb [20:49:12] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb [20:52:24] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 4.663 second response time [20:53:39] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:54:53] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:40] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:55:55] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:57:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvXBd [20:57:38] [02miraheze/mw-config] 07paladox 03d965c3a - Increase wgHTTPTimeout and wgHTTPConnectTimeout to 60 [20:57:39] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [20:57:41] [02mw-config] 07paladox opened pull request 03#2945: Increase wgHTTPTimeout and wgHTTPConnectTimeout to 60 - 13https://git.io/JvXBF [20:57:48] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:58:02] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:58:18] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw5 [20:58:27] PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:58:32] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw5 [20:58:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [20:58:47] paladox: ^ [20:58:58] yes i am aware [20:59:12] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 1.742 second response time [20:59:13] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw7 [20:59:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvXBN [20:59:19] [02miraheze/puppet] 07paladox 031cfbb56 - parsoid: Decrease timeout [20:59:51] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:10] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:31] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 8.976 second response time [21:02:35] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [21:02:35] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [21:02:54] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.753 second response time [21:03:07] RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.306 second response time [21:03:41] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [21:04:03] PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:04:27] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 18694 bytes in 0.700 second response time [21:04:27] !log reboot services1 [21:04:36] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.309 second response time [21:04:39] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18720 bytes in 2.377 second response time [21:04:40] !log reboot services2 [21:04:51] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 1.646 second response time [21:04:54] !log restart php7.3-fpm on mw* [21:05:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:05:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:06:36] paladox: Save successful. Did you change something? [21:06:51] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [21:06:57] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [21:06:59] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [21:07:43] RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.357 second response time [21:08:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:08:57] I reduced the timeouts for parsoid and restarted parsoid [21:09:19] Thanks! [21:29:59] Hello malexander! If you have any questions, feel free to ask and someone should answer soon. [21:33:46] How can we help? [21:39:07] PartyRhino it's MikeV [21:39:54] that's true [21:40:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvXRS [21:40:09] [02miraheze/services] 07MirahezeSSLBot 0387f3a96 - BOT: Updating services config for wikis [22:02:38] Yep. Trying a different IRC client. [22:06:32] Appreciate the offer, though. I would really like to be able to use the "Download as PDF" on our private ndg.nenawiki.org site. Is this a miraheze setting, or is it an upstream limitation? [22:18:13] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:24:52] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.437 second response time [22:25:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [22:26:50] yey, nothing broke! [22:32:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:36:07] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%);