[00:02:31] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3564 MB (18% inode=93%);
[00:22:16] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVpR
[00:22:17] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 0383e3ff4 - Bot: Update SSL cert for wiki.iufs.cf
[00:22:29] <paladox>	 \o/
[00:22:55] <paladox>	 oh
[00:26:30] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVpE
[00:26:31] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 03a23bae1 - Bot: Update SSL cert for wiki.iufs.cf
[00:29:48] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures
[00:32:35] <Not-5a0a>	 [02mw-config] 07Hispano76 opened pull request 03#2927: wgNamespaceRobotPolicies on hispanowiki and ucronias - 13https://git.io/JvVpr
[00:36:31] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVpK
[00:36:32] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 036cddaba - Bot: Update SSL cert for wiki.iufs.cf
[00:36:42] <paladox>	 \o/
[00:37:26] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1274: varnish: Switch backend to jobrunner1 for .well-known - 13https://git.io/JvVA3
[00:37:28] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvVp6
[00:37:29] <Not-5a0a>	 [02miraheze/puppet] 07paladox 033db3efd - varnish: Switch backend to jobrunner1 for .well-known (#1274) * varnish: Switch backend to jobrunner1 for .well-known * Update stunnel.conf * Update default.vcl
[00:37:31] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[00:37:32] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[00:40:13] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVpP
[00:40:14] <Not-5a0a>	 [02miraheze/puppet] 07paladox 035550920 - monitoring: Switch to jobrunner1 as the backend to use for renewing certs
[00:53:29] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVpQ
[00:53:31] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 03adacd9f - Bot: Update SSL cert for thelonsdalebattalion.co.uk
[00:54:17] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVp7
[00:54:18] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 03d415135 - Bot: Update SSL cert for thelonsdalebattalion.co.uk
[00:59:49] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVhe
[00:59:51] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 033e020f5 - Bot: Update SSL cert for thelonsdalebattalion.co.uk
[01:02:13] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVhU
[01:02:14] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 038310244 - Bot: Update SSL cert for wiki.vinesh.eu.org
[01:05:07] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVhL
[01:05:08] <Not-5a0a>	 [02miraheze/puppet] 07paladox 039b116f1 - varnish: Add test2
[01:05:10] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[01:05:11] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1275: varnish: Add test2 - 13https://git.io/JvVht
[01:05:50] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVhm
[01:05:52] <Not-5a0a>	 [02miraheze/puppet] 07paladox 038124900 - Update default.vcl
[01:05:53] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1275: varnish: Add test2 - 13https://git.io/JvVht
[01:06:14] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVhY
[01:06:15] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03be6d9e7 - Update nrpe.cfg.erb
[01:06:17] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1275: varnish: Add test2 - 13https://git.io/JvVht
[01:06:39] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVhO
[01:06:40] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03185113c - Update nrpe.cfg.erb
[01:06:42] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1275: varnish: Add test2 - 13https://git.io/JvVht
[01:07:29] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVh3
[01:07:31] <Not-5a0a>	 [02miraheze/puppet] 07paladox 032e1bc91 - Update init.pp
[01:07:32] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1275: varnish: Add test2 - 13https://git.io/JvVht
[01:07:53] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1275: varnish: Add test2 - 13https://git.io/JvVht
[01:07:54] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JvVhG
[01:07:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03ae17a97 - varnish: Add test2 (#1275)
[01:08:58] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[01:08:59] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[01:11:20] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for test2 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.001 second response time
[01:11:23] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[01:12:39] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for test2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 11043 bytes in 0.002 second response time
[01:12:42] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for test2 on cp8 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.002 second response time
[01:13:09] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.502 second response time
[01:13:10] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mon1 on cp8 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:15] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mon1 on cp4 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:17] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 8 backends are down. mw1 mw2 mw3 lizardfs6 mw4 mw5 mw6 mw7
[01:13:20] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:23] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.536 second response time
[01:13:24] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.564 second response time
[01:13:26] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:31] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 80%
[01:13:41] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 97%
[01:13:42] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw6 on cp4 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:43] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 8 backends are down. mw1 mw2 mw3 lizardfs6 mw4 mw5 mw6 mw7
[01:13:47] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:49] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:13:50] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 8 backends are down. mw1 mw2 mw3 lizardfs6 mw4 mw5 mw6 mw7
[01:13:51] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVhl
[01:13:53] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c5343a8 - Update stunnel.conf
[01:13:54] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[01:13:56] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1276: Update stunnel.conf - 13https://git.io/JvVh8
[01:13:59] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: connect to address localhost and port 8087: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:05] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mon1 on cp3 is CRITICAL: connect to address localhost and port 8201: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:11] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:18] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvVh4
[01:14:19] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03086a282 - Update default.vcl
[01:14:21] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1276: Update stunnel.conf - 13https://git.io/JvVh8
[01:14:23] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw7 on cp4 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:28] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1276: Update stunnel.conf - 13https://git.io/JvVh8
[01:14:29] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvVhB
[01:14:31] <Not-5a0a>	 [02miraheze/puppet] 07paladox 036cde93b - Update stunnel.conf (#1276)
[01:14:35] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: connect to address localhost and port 8088: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:36] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw5 on cp4 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:37] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[varnish-server-syntax],Service[stunnel4]
[01:14:39] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: connect to address localhost and port 8086: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:14:47] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw4 on cp4 is CRITICAL: connect to address localhost and port 8085: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:15:10] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19676 bytes in 0.424 second response time
[01:15:17] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[varnish-server-syntax],Service[stunnel4]
[01:15:18] <icinga-miraheze>	 RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19676 bytes in 0.422 second response time
[01:15:20] <icinga-miraheze>	 RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19676 bytes in 0.516 second response time
[01:15:43] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.791 second response time
[01:15:49] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[01:15:50] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.769 second response time
[01:15:50] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[01:16:01] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mon1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 29527 bytes in 1.018 second response time
[01:16:08] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.748 second response time
[01:16:35] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.761 second response time
[01:16:39] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.320 second response time
[01:16:53] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[varnish-server-syntax],Service[stunnel4]
[01:17:03] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mon1 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 29496 bytes in 0.320 second response time
[01:17:19] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.308 second response time
[01:17:24] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.323 second response time
[01:17:32] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 56%
[01:17:34] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw6 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.117 second response time
[01:17:35] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 2%
[01:17:43] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 10 backends are healthy
[01:17:47] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy
[01:17:53] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVh0
[01:17:55] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0342f9dcb - fix
[01:17:57] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.311 second response time
[01:18:12] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw7 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.085 second response time
[01:18:37] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw5 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.080 second response time
[01:18:42] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw4 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.079 second response time
[01:18:50] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:19:14] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy
[01:19:15] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[01:19:17] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mon1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 29496 bytes in 0.088 second response time
[01:19:26] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[01:19:33] <icinga-miraheze>	 RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 1%
[01:20:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVhu
[01:20:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03e41f3d2 - BOT: Updating services config for wikis
[01:20:32] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:23:30] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVhV
[01:23:32] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03157437c - mw1: Disable letsencrypt
[01:24:36] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:26:02] <icinga-miraheze>	 PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: connect to address 185.52.1.75 and port 5000: Connection refused
[01:28:13] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for lizardfs6 on cp4 is CRITICAL: connect to address localhost and port 8203: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:28:22] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for lizardfs6 on cp8 is CRITICAL: connect to address localhost and port 8203: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:28:34] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for lizardfs6 on cp3 is CRITICAL: connect to address localhost and port 8203: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:32:59] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvVhM
[01:33:00] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0364e27cb - Update nrpe.cfg.erb
[02:17:04] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21980 MB (5% inode=95%);
[06:26:28] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3269 MB (13% inode=94%);
[09:41:10] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[09:45:17] <icinga-miraheze>	 RECOVERY - udc-japan.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'udc-japan.xyz' will expire on Wed 22 Apr 2020 11:07:34 PM GMT +0000.
[09:52:45] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[09:54:45] <icinga-miraheze>	 RECOVERY - udc-japan.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'udc-japan.xyz' will expire on Wed 22 Apr 2020 11:07:34 PM GMT +0000.
[10:10:06] <icinga-miraheze>	 PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[13:28:23] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwBW
[13:28:24] <Not-5a0a>	 [02miraheze/puppet] 07paladox 039b64426 - Update config.ini.php.erb
[14:01:45] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvwR3
[14:01:47] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03f782651 - Set all large wikis into read only
[14:01:48] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[14:01:50] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2928: Set all large wikis into read only - 13https://git.io/JvwRs
[14:05:23] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwRC
[14:05:24] <Not-5a0a>	 [02miraheze/dns] 07paladox 030020fda - Switch all large wikis to use cp7
[14:05:26] <Not-5a0a>	 [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl
[14:05:27] <Not-5a0a>	 [02dns] 07paladox opened pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[14:05:42] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[14:05:43] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwR4
[14:05:45] <Not-5a0a>	 [02miraheze/dns] 07paladox 03d6f187c - Update bebaskanpengetahuan.id
[14:09:04] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwR2
[14:09:05] <Not-5a0a>	 [02miraheze/dns] 07paladox 03db33f20 - Update allthetropes.org
[14:09:07] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[14:10:19] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwRV
[14:10:21] <Not-5a0a>	 [02miraheze/dns] 07paladox 037b901ec - Update nonciclopedia.org
[14:10:22] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[14:15:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwRD
[14:15:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03ff21069 - BOT: Updating services config for wikis
[14:16:09] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwRS
[14:16:11] <Not-5a0a>	 [02miraheze/dns] 07paladox 03ef8d85c - Update miraheze.org
[14:16:12] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[14:26:19] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jvw0I
[14:26:21] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 039d9be0f - database: Switch all large wikis to db6
[14:26:22] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3
[14:26:24] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2929: database: Switch all large wikis to db6 - 13https://git.io/Jvw0L
[14:28:39] <Zppix>	 Just a reminder the wikis we consider large (usually over 1gb) will be migrated today at 7pm UTC (12pm UTC-5)
[16:01:05] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvwg7
[16:01:07] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03bda11c3 - prometheus::mysqld_exporter: Add mon1 to firewall
[16:03:35] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwgF
[16:03:36] <Not-5a0a>	 [02miraheze/puppet] 07paladox 034402ad2 - prometheus::gluster_exporter: Add mon1 to firewall
[16:04:26] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvwgb
[16:04:28] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03b5f454d - prometheus: Ensure nginx-prometheus-exporter service is running
[16:07:19] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwgA
[16:07:20] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03ccc1363 - Update nginx-prometheus-exporter to 0.6.0
[16:12:07] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:10] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvw2q
[16:12:12] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03755f075 - Update nginx.pp
[16:12:35] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:45] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:12:57] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:17] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:29] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:33] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:44] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:13:47] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvw2G
[16:13:49] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0331704bd - Update nginx.pp
[16:14:05] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[16:14:07] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:16:10] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[16:21:43] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[16:22:29] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[16:22:46] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:22:54] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvw2r
[16:22:55] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03fb09dfb - Update redis_exporter to 1.4.0
[16:23:00] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:23:18] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:23:32] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[16:23:35] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:24:26] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvw2i
[16:24:27] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03e3bf86f - Update redis_exporter.pp
[16:28:48] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%);
[16:30:01] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvw2b
[16:30:02] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0329b0956 - Fix fetching all hosts that have redis installed
[16:30:04] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[16:30:05] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1277: Fix fetching all hosts that have redis installed - 13https://git.io/Jvw2N
[16:33:44] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvwav
[16:33:45] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03d12e20e - Update prometheus.yml.erb
[16:33:47] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1277: Fix fetching all hosts that have redis installed - 13https://git.io/Jvw2N
[16:34:02] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1277: Fix fetching all hosts that have redis installed - 13https://git.io/Jvw2N
[16:34:04] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvwaJ
[16:34:05] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03a799a00 - Fix fetching all hosts that have redis installed (#1277) * Fix fetching all hosts that have redis installed * Update prometheus.yml.erb
[16:34:07] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[16:34:08] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[16:35:17] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwaU
[16:35:18] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03f7135b1 - Fix fetching all hosts that have mariadb installed
[16:35:20] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[16:35:21] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1278: Fix fetching all hosts that have mariadb installed - 13https://git.io/JvwaT
[16:36:11] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvwat
[16:36:13] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c66db7c - Update prometheus.yml.erb
[16:36:14] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1278: Fix fetching all hosts that have mariadb installed - 13https://git.io/JvwaT
[16:36:44] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1278: Fix fetching all hosts that have mariadb installed - 13https://git.io/JvwaT
[16:36:46] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jvwaq
[16:36:47] <Not-5a0a>	 [02miraheze/puppet] 07paladox 033c1a4c1 - Fix fetching all hosts that have mariadb installed (#1278) * Fix fetching all hosts that have mariadb installed * Update prometheus.yml.erb
[16:36:48] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[16:36:50] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[16:43:06] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[16:46:34] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.448 second response time
[16:46:38] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 4 backends are down. mw1 mw2 mw3 lizardfs6
[16:46:40] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw1 mw2 mw3 lizardfs6
[16:47:31] <Reception123>	 ^ paladox 
[16:47:37] <paladox>	 I'm thinking that's redis
[16:47:57] <paladox>	 should be back now
[16:49:19] <Reception123>	 ok
[16:50:30] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy
[16:50:31] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 10 backends are healthy
[16:51:13] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2114 MB (10% inode=93%);
[16:52:00] <icinga-miraheze>	 PROBLEM - wiki.hrznstudio.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.hrznstudio.com' expires in 15 day(s) (Sun 22 Mar 2020 04:43:33 PM GMT +0000).
[16:52:29] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19645 bytes in 2.748 second response time
[16:52:34] <paladox>	 !log upgrade puppet-agent on misc4
[16:52:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:52:53] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:58:53] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwaX
[16:58:55] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 0374a2a4c - Bot: Update SSL cert for wiki.hrznstudio.com
[17:03:39] <icinga-miraheze>	 RECOVERY - wiki.hrznstudio.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hrznstudio.com' will expire on Thu 04 Jun 2020 03:58:47 PM GMT +0000.
[17:06:34] <Zppix>	 Its a shame its so quiet in here... must of scared all the trolls away Reception123
[17:09:12] <Reception123>	 Zppix: definitely wouldn't call it a shame...
[17:09:22] <Reception123>	 the GH bot has been annoying enough 
[17:09:26] <Zppix>	 True
[17:09:32] <Reception123>	 Zppix: and don't say that or you'l tempt icinga 
[17:09:44] <Zppix>	 Haha
[17:30:15] <Zppix>	 We will be migrating large wikis in 30 minutes! Please keep in mind wikis listed will be set in read-only. If you have any questions feel free to reach out to member of staff! https://meta.miraheze.org/wiki/User:Paladox/Migration_2020-03-06_list
[17:30:16] <ZppixBot>	 [ User:Paladox/Migration 2020-03-06 list - Miraheze Meta ] - meta.miraheze.org
[17:30:26] <paladox>	 Zppix wrong time.
[17:30:31] <paladox>	 Zppix 1hr and 30 mins
[17:30:42] <Zppix>	 .t utc
[17:30:43] <ZppixBot>	 2020-03-06 - 17:30:42UTC
[17:30:47] <Zppix>	 Smh
[17:31:48] <Zppix>	 I suck at time zone conversions
[18:14:59] <k6ka>	 timeanddate.com is king
[18:15:58] <Zppix>	 Sssh
[18:16:53] <RhinosF1>	 k6ka: agreed
[18:35:27] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[18:35:28] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvwrW
[18:35:30] <Not-5a0a>	 [02miraheze/dns] 07paladox 036f3775a - Update allthetropes.org
[18:35:42] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[18:35:44] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvwrl
[18:35:45] <Not-5a0a>	 [02miraheze/dns] 07paladox 030440d33 - Update bebaskanpengetahuan.id
[18:36:02] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[18:36:03] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvwr8
[18:36:05] <Not-5a0a>	 [02miraheze/dns] 07paladox 03dd902c0 - Update miraheze.org
[18:36:17] <Not-5a0a>	 [02dns] 07paladox synchronize pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[18:36:19] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvwr4
[18:36:21] <Not-5a0a>	 [02miraheze/dns] 07paladox 037ef9036 - Update nonciclopedia.org
[18:38:21] <paladox>	 !log sysctl -w net.ipv6.conf.all.disable_ipv6=1 - db5
[18:38:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:43:12] <RhinosF1>	 paladox: good luck! I’m here all night. If there are any time changes, let me know.
[18:43:31] <paladox>	 thanks!
[18:44:46] <RhinosF1>	 Np!
[18:46:08] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvwr6
[18:46:10] <Not-5a0a>	 [02miraheze/puppet] 07paladox 030ef64e2 - Update site.pp
[18:57:10] <Not-5a0a>	 [02dns] 07paladox closed pull request 03#138: Switch all large wikis to use cp7 - 13https://git.io/JvwRl
[18:57:11] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jvwrb
[18:57:13] <Not-5a0a>	 [02miraheze/dns] 07paladox 036636781 - Switch all large wikis to use cp7 (#138)
[18:57:14] <Not-5a0a>	 [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl
[18:57:16] <Not-5a0a>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2
[18:59:16] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2928: Set all large wikis into read only - 13https://git.io/JvwRs
[18:59:17] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwrA
[18:59:19] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0313d3bf4 - Set all large wikis into read only (#2928)
[18:59:20] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[18:59:22] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[19:02:20] <Zppix>	 !log Migration window has started
[19:02:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:03:19] <paladox>	 !log root@db4:/home/paladox# ./move_all.sh
[19:03:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:28:20] <paladox>	 !log root@db5:/home/paladox# ./move_all.sh
[20:28:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:28:39] <RhinosF1>	 paladox: how we going?
[20:28:57] <paladox>	 db4 done, doing db5 now
[20:29:17] <RhinosF1>	 paladox: what percent is done approx then?
[20:29:26] <paladox>	 Percent?
[20:29:52] <Voidwalker>	 how much space is that, out of curiosity?
[20:29:57] <RhinosF1>	 paladox: how many gig on db4 and how many on db5?
[20:30:21] <SPF|Cloud>	 paladox, RhinosF1: hi, good work :)
[20:30:32] <paladox>	 Oh, i haven't counted. I just ran a quick command to find all the dbs that are >=1gb
[20:30:36] <paladox>	 SPF|Cloud thanks!
[20:30:55] <paladox>	 db5 has a smaller list to migrate, but the dbs are bigger in terms of size
[20:31:34] <RhinosF1>	 paladox: okay
[20:31:38] <paladox>	 Voidwalker RhinosF1 i can give you an estimate for db5
[20:31:49] <RhinosF1>	 That’s goos
[20:31:57] <RhinosF1>	 paladox: pls
[20:32:03] <paladox>	 ~97G
[20:32:34] <paladox>	 oh
[20:32:41] <paladox>	 wait, wrong column
[20:32:43] <paladox>	 RhinosF1 Voidwalker ~82G
[20:33:28] <RhinosF1>	 paladox: can you do the same on db4
[20:33:39] <paladox>	 db4 won't be easy
[20:34:08] <paladox>	 since db5 was storing mostly the large wikis rather then a mix. So that was an easy calc
[20:34:22] <RhinosF1>	 Ah
[20:35:02] <RhinosF1>	 paladox: what speed is it going at?
[20:35:30] <paladox>	 i have no idea :), it's just running the script without outputting anything.
[20:35:43] <paladox>	 i'm using `show processlist` to know where it's at
[20:36:03] <RhinosF1>	 Ah
[20:36:37] <paladox>	 it's on simswiki
[20:37:26] <RhinosF1>	 What order does it go?
[20:37:34] <paladox>	 random order.
[20:37:44] <paladox>	 i mean it's doing it in the order i gave
[20:37:52] <paladox>	 but it's random (not in alphabetical order)
[20:38:12] <Voidwalker>	 not terribly useful to know where it is at then is it
[20:38:32] <RhinosF1>	 paladox: what order did you give it then?
[20:38:46] <paladox>	 https://etherpad.wikimedia.org/p/zv81kYuPnWa3T2brEc0r
[20:38:46] <ZppixBot>	 [ Etherpad ] - etherpad.wikimedia.org
[20:51:50] <RhinosF1>	 Hi SPF|Cloud, Thanks! I’ve got the easy task here. Enjoy the board meeting.
[20:52:00] <SPF|Cloud>	 we do our best
[20:52:21] <RhinosF1>	 SPF|Cloud: It’s going well!
[20:52:29] <SPF|Cloud>	 and easy, well... if communication was such an easy task, we wouldn't have all these advisors in businesses
[20:53:14] <RhinosF1>	 True, It’s our users that make it easy by being so nice!
[20:54:44] <SPF|Cloud>	 please do your thing, after all it's about collaboration between teams/disciplines :-)
[20:55:06] <RhinosF1>	 It certainly is
[20:59:07] <RhinosF1>	 paladox: how we going as we hit the halfway mark?
[21:01:13] <paladox>	 It's on allthetropeswiki
[21:05:17] <RhinosF1>	 paladox: that’s speedy! These should be our biggest though!
[21:30:34] <RhinosF1>	 paladox: what wiki?
[21:30:44] <paladox>	 still on att
[21:30:53] <paladox>	 it's ~30gb
[21:31:01] <RhinosF1>	 Yeah True
[21:31:46] <Zppix>	 Att probably will take up to 30mins+ (i would assume but im probably wrong)
[21:33:33] <RhinosF1>	 My maths says that if db5 takes the full 2 hours left from when it started then it should be ~45 mins
[21:34:05] <RhinosF1>	 As it’s 30/82 is 36% - 36% of 120 mins is 43
[21:34:14] <RhinosF1>	 paladox: ^ that sound right
[21:34:38] <paladox>	 It took ~40 minutes the last time i tried
[21:35:13] <RhinosF1>	 Probably then
[21:41:55] <paladox>	 it's now on nonciclopediawiki
[21:42:01] <paladox>	 that's ~20g
[21:43:26] <RhinosF1>	 paladox: yey!
[22:14:15] <paladox>	 it's on toxicfandomsandhatedomswiki now
[22:14:49] <paladox>	 it's on nonsensopediawiki now
[22:15:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvw1s
[22:15:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03eea67fd - BOT: Updating services config for wikis
[22:15:32] <RhinosF1>	 paladox: fantastic!
[22:20:58] <RhinosF1>	 paladox: do we know how big this db is?
[22:21:28] <paladox>	 11g
[22:29:13] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[22:33:23] <paladox>	 We're finished!
[22:33:37] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2929: database: Switch all large wikis to db6 - 13https://git.io/Jvw0L
[22:33:38] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvw1N
[22:33:40] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 034730cb4 - database: Switch all large wikis to db6 (#2929)
[22:33:41] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3
[22:33:43] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3
[22:34:06] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[22:38:19] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvwMT
[22:38:21] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 037787c78 - Take large wikis out of read only
[22:38:22] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[22:38:24] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2930: Take large wikis out of read only - 13https://git.io/JvwMk
[22:40:02] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2930: Take large wikis out of read only - 13https://git.io/JvwMk
[22:40:04] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwMC
[22:40:05] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 031b6de06 - Take large wikis out of read only (#2930)
[22:40:07] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[22:40:08] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[22:41:20] <Examknow>	 paladox: yayyy! :)
[22:41:54] <paladox>	 :)
[22:42:39] <RhinosF1>	 paladox: Well Done! That was fantastic! I’ll stay around for a bit longer. Make sure you get rest before Sunday.
[22:42:51] <paladox>	 thanks :)
[22:44:52] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvwMB
[22:44:54] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 035413981 - Remove site notice from large wikis
[22:44:55] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[22:44:57] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2931: Remove site notice from large wikis - 13https://git.io/JvwMR
[22:45:08] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2931: Remove site notice from large wikis - 13https://git.io/JvwMR
[22:45:10] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwME
[22:45:11] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03798292b - Remove site notice from large wikis (#2931)
[22:45:12] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[22:45:14] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[22:46:27] <travis-ci>	 miraheze/mw-config/paladox-patch-1/5413981 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/659365762
[22:58:23] <RhinosF1>	 paladox: can you set the topic back to Status: OK?
[22:58:41] <paladox>	 done
[22:58:52] <RhinosF1>	 :)
[23:35:32] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4237 bytes in 0.425 second response time
[23:37:28] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19644 bytes in 1.064 second response time
[23:50:42] <SPF|Cloud>	 !log run pveperf on cloud1
[23:50:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master