[00:01:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:37:44] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 19.44, 13.86, 6.84 [00:38:05] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:19] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:38:40] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:49] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:57] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [00:39:05] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [00:39:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:39:37] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1367 bytes in 0.036 second response time [00:40:04] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:40:28] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18579 bytes in 3.907 second response time [00:40:33] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18579 bytes in 0.915 second response time [00:40:40] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.691 second response time [00:40:58] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24668 bytes in 4.176 second response time [00:41:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [00:41:31] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [00:41:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:41:54] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18585 bytes in 0.370 second response time [00:42:01] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.007 second response time [00:42:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:43:14] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.79, 2.77, 1.83 [00:45:13] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.37, 1.46, 1.50 [00:47:56] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 1.85, 6.08, 7.23 [00:49:52] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 2.47, 4.99, 6.70 [00:54:16] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],Service[ssh],Package[exim4] [01:00:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JeNOr [01:00:24] [02miraheze/puppet] 07paladox 032e0a2f7 - redis: Disable persistence [01:00:26] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS [01:00:27] [02puppet] 07paladox opened pull request 03#1169: redis: Disable persistence - 13https://git.io/JeNOo [01:01:29] [02puppet] 07JohnFLewis closed pull request 03#1169: redis: Disable persistence - 13https://git.io/JeNOo [01:01:30] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNO6 [01:01:32] [02miraheze/puppet] 07paladox 03eacf15e - redis: Disable persistence (#1169) [01:02:01] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [01:29:56] RECOVERY - db5 Disk Space on db5 is OK: DISK OK - free space: / 20108 MB (11% inode=99%); [01:33:02] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:33:52] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:12] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:12] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:23] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:48] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:59] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:35:01] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [01:35:42] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 13% [01:37:33] PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 20057 MB (10% inode=99%); [01:38:03] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:28] !log apt-get upgrade - cp3 [01:39:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:40:41] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.861 second response time [01:41:05] !log apt-get install linux-image-amd64 [01:41:06] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.691 second response time [01:41:09] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1532 bytes in 1.738 second response time [01:41:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:41:31] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24668 bytes in 0.665 second response time [01:42:05] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeN3v [01:42:07] [02miraheze/dns] 07paladox 03a001517 - Depool cp3 [01:42:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:51] !log reboot cp3 [01:44:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:45:46] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [01:45:51] PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:57] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:46:17] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:46:30] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 5 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [01:46:57] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: connect to address 128.199.139.216 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:47:06] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:18] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:33] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeN3L [01:47:35] [02miraheze/dns] 07paladox 038bcb4b9 - Revert "Depool cp3" This reverts commit a001517da5c3cbb30c6dcfdc436ca66bbe6d1bd0. [01:47:57] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:47:58] RECOVERY - cp3 Stunnel Http for test1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.845 second response time [01:48:05] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2534 MB (10% inode=94%); [01:48:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:48:17] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.919 second response time [01:48:53] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1515 bytes in 0.808 second response time [01:49:04] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.684 second response time [01:49:06] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 27% [01:49:17] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.682 second response time [01:49:17] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.41, 0.35, 0.15 [01:49:44] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [01:49:55] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [01:52:23] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:43:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [03:51:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [04:03:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [04:11:40] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNGN [05:40:09] [02miraheze/services] 07MirahezeSSLBot 036778965 - BOT: Updating services config for wikis [06:27:19] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3434 MB (14% inode=94%); [07:53:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [08:01:41] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [09:33:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [09:41:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [11:18:52] [02landing] 07RhinosF1 opened pull request 03#20: Update paypal info - 13https://git.io/JeNCc [11:19:46] paladox, Reception123: ^ [11:21:20] Don't have GH access now [11:21:30] If paladox doesn't get to it I'll be able to in about 4 hours [11:22:18] Reception123: k [12:07:57] RELAY IS ON A XMAS BREAK UNTIL JAN 1 [12:09:10] Reception123: I can merge it [12:09:19] but I dont know if i have access to make sure it deploys [12:10:12] Zppix: you can merge, just wait for puppet [12:10:59] [02landing] 07Pix1234 closed pull request 03#20: Update paypal info - 13https://git.io/JeNCc [12:11:00] [02miraheze/landing] 07Pix1234 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JeNCj [12:11:01] [02miraheze/landing] 07RhinosF1 038f11c0e - Update paypal info Now in GBP not USD [12:11:03] [02miraheze/landing] 07Pix1234 0310d4d6b - Merge pull request #20 from RhinosF1/patch-1 Update paypal info [12:11:08] Zppix: it's only grammar so it's fine I guess [12:11:18] Tho I dont know why mw admins have access to merge [12:11:27] Reception123: why not :P [12:12:06] Reception123: makes no sense but it’s given [12:12:37] seeing how im the only mw-admin atm *shrug* [12:13:16] * Zppix is trustworthy... sometimes [12:13:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [12:13:44] ssh its okay icinga-miraheze [12:15:49] Zppix: looks like it’s OOMing [12:17:07] it is iirc [12:17:49] Zppix: all the bots are moody today - even wm-bot550 won’t log in to ns [12:21:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [12:22:24] Zppix: ^ back and email cloud mailing list on the thread to kick wm-bot550 back into play [12:22:32] Emailed* [12:33:40] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [12:41:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:05:06] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNWx [14:05:08] [02miraheze/services] 07MirahezeSSLBot 03e8ec3e8 - BOT: Updating services config for wikis [14:19:37] PROBLEM - or.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:19:57] PROBLEM - fa.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:06] PROBLEM - meta.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:10] PROBLEM - ml.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:17] PROBLEM - sa.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:19] PROBLEM - hi.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:24] PROBLEM - mg.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:28] PROBLEM - mai.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:30] PROBLEM - mr.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:38] PROBLEM - gu.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:41] PROBLEM - te.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:51] PROBLEM - commons.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:20:54] PROBLEM - pa.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:06] PROBLEM - la.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:12] PROBLEM - bn.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:18] PROBLEM - hif.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:23] PROBLEM - ta.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:27] PROBLEM - en.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:30] PROBLEM - kn.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.gyaanipedia.co.in' expires in 15 day(s) (Mon 13 Jan 2020 02:17:34 PM GMT +0000). [14:21:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNlY [14:21:51] [02miraheze/ssl] 07MirahezeSSLBot 036b15910 - Bot: Update SSL cert for en.gyaanipedia.co.in [14:32:04] RECOVERY - meta.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:10] RECOVERY - ml.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:17] RECOVERY - sa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:19] RECOVERY - hi.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:26] RECOVERY - mai.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:33] RECOVERY - mr.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:35] RECOVERY - mg.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:40] RECOVERY - gu.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:43] RECOVERY - te.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:46] RECOVERY - pa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:51] RECOVERY - commons.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:32:58] RECOVERY - la.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:10] RECOVERY - bn.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:17] RECOVERY - hif.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:22] RECOVERY - ta.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:25] RECOVERY - en.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:29] RECOVERY - kn.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:37] RECOVERY - or.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:33:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [14:33:57] RECOVERY - fa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Fri 27 Mar 2020 01:21:43 PM GMT +0000. [14:51:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [15:43:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [15:51:39] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [15:52:11] RhinosF1: by the way tell me when you'd want to test Twinkle. And you can get test1 user/admin access to help [15:57:29] Reception123: i’ve got friday free [16:00:23] ok [16:00:39] I'll try to see if I can do anything before and then we'll see but it's not going to be easy [16:00:49] Reception123: twinkle isnt easy [16:32:27] * hispano76 greeting [16:33:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [16:41:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [17:26:32] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:24] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:27:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:27:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [17:27:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [17:28:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:28:36] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24650 bytes in 0.009 second response time [17:29:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:29:24] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24650 bytes in 0.673 second response time [17:29:35] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [17:29:38] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [17:30:04] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:34:42] PROBLEM - www.modesofdiscourse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.modesofdiscourse.com' expires in 15 day(s) (Mon 13 Jan 2020 05:32:31 PM GMT +0000). [17:34:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNRn [17:34:56] [02miraheze/ssl] 07MirahezeSSLBot 0394a7a74 - Bot: Update SSL cert for www.modesofdiscourse.com [17:42:26] RECOVERY - www.modesofdiscourse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.modesofdiscourse.com' will expire on Fri 27 Mar 2020 04:34:48 PM GMT +0000. [18:15:56] PROBLEM - www.jawright.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.jawright.cc' expires in 15 day(s) (Mon 13 Jan 2020 06:12:57 PM GMT +0000). [18:16:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeN0n [18:16:11] [02miraheze/ssl] 07MirahezeSSLBot 038e73938 - Bot: Update SSL cert for www.jawright.cc [18:17:06] PROBLEM - jawright.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.jawright.cc' expires in 15 day(s) (Mon 13 Jan 2020 06:12:57 PM GMT +0000). [18:23:15] RECOVERY - jawright.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'www.jawright.cc' will expire on Fri 27 Mar 2020 05:16:04 PM GMT +0000. [18:23:52] RECOVERY - www.jawright.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'www.jawright.cc' will expire on Fri 27 Mar 2020 05:16:04 PM GMT +0000. [19:53:40] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],Service[ssh],Package[exim4] [20:01:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:15:33] JohnLewis, Reception123: can you answer https://phabricator.miraheze.org/T5034#95822 [20:15:34] [ ⚓ T5034 Change database name and domain ] - phabricator.miraheze.org [20:16:40] No access atm [20:17:40] Reception123: they want it done during a time of the day we’ve missed for today anyway. Just want you to answet theit questions [20:18:32] https://phabricator.miraheze.org/T5032 As well needs doing fairly soon [20:18:33] [ ⚓ T5032 Request for Custom Domain ] - phabricator.miraheze.org [20:29:56] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeNze [20:29:58] [02miraheze/ssl] 07paladox 033cd02f9 - Add vedopedia.witches-empire.com ssl certificate [20:29:59] [02ssl] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vxP9L [20:30:01] [02ssl] 07paladox opened pull request 03#250: Add vedopedia.witches-empire.com ssl certificate - 13https://git.io/JeNzv [20:30:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JeNzU [20:30:35] [02miraheze/ssl] 07paladox 0364b0f74 - Create vedopedia.witches-empire.com.crt [20:30:37] [02ssl] 07paladox synchronize pull request 03#250: Add vedopedia.witches-empire.com ssl certificate - 13https://git.io/JeNzv [20:31:05] [02ssl] 07paladox closed pull request 03#250: Add vedopedia.witches-empire.com ssl certificate - 13https://git.io/JeNzv [20:31:06] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeNzT [20:31:08] [02miraheze/ssl] 07paladox 0327b97f6 - Add vedopedia.witches-empire.com ssl certificate (#250) * Add vedopedia.witches-empire.com ssl certificate * Create vedopedia.witches-empire.com.crt [20:31:17] [02ssl] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vxP9L [20:31:18] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-2 [20:34:35] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeNzs [20:34:37] [02miraheze/landing] 07paladox 033d24d9f - Redesign landing page [20:34:38] [02landing] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fxEQA [20:34:40] [02landing] 07paladox opened pull request 03#21: Redesign landing page - 13https://git.io/JeNzG [20:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNzc [20:35:08] [02miraheze/services] 07MirahezeSSLBot 030e9aac6 - BOT: Updating services config for wikis [20:35:27] miraheze/landing/paladox-patch-1/3d24d9f - paladox The build passed. https://travis-ci.org/miraheze/landing/builds/630427036 [20:53:06] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JeNzV [20:53:07] [02miraheze/mediawiki] 07paladox 0376ffc7a - Update DiscordNotifications [21:13:07] [02landing] 07paladox synchronize pull request 03#21: Redesign landing page - 13https://git.io/JeNzG [21:13:08] [02miraheze/landing] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeNzQ [21:13:10] [02miraheze/landing] 07paladox 03e9f88e7 - Update index.php [21:23:24] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [21:33:24] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:43:24] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [21:45:32] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNgb [21:45:33] [02miraheze/puppet] 07JohnFLewis 034b70d5e - assign alias campaign to sre [21:46:40] Ooooh [21:46:49] That sounds good [22:23:24] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:43:01] !log rebuild lc on mw* and lizard [22:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:47:34] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:48:36] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:49:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [22:49:38] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24672 bytes in 8.400 second response time [22:52:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [22:53:23] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:53:56] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:57] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:55:19] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24650 bytes in 3.725 second response time [22:55:53] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18584 bytes in 0.269 second response time [22:59:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JeNar [22:59:20] [02miraheze/puppet] 07paladox 0386e93cf - Redis: Disable RDB and AOF [22:59:21] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS [22:59:23] [02puppet] 07paladox opened pull request 03#1170: Redis: Disable RDB and AOF - 13https://git.io/JeNao [23:01:01] [02puppet] 07paladox synchronize pull request 03#1170: Redis: Disable RDB and AOF - 13https://git.io/JeNao [23:01:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JeNai [23:01:04] [02miraheze/puppet] 07paladox 03ecf6186 - Update redis.conf.erb [23:01:09] [02puppet] 07paladox closed pull request 03#1170: Redis: Disable RDB and AOF - 13https://git.io/JeNao [23:01:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNaP [23:01:12] [02miraheze/puppet] 07paladox 031adcbbd - Redis: Disable RDB and AOF (#1170) * Redis: Disable RDB and AOF * Update redis.conf.erb [23:01:13] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://git.io/vbiAS [23:01:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [23:01:35] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [23:03:41] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:13:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [23:13:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:15:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:15:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:21:29] [02mw-config] 07jidanni opened pull request 03#2841: Update ManageWikiSettings.php - 13https://git.io/JeNab [23:24:54] [02mw-config] 07paladox closed pull request 03#2841: Update ManageWikiSettings.php - 13https://git.io/JeNab [23:24:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeNVJ [23:24:57] [02miraheze/mw-config] 07jidanni 031ff1f90 - Update ManageWikiSettings.php (#2841) Explain what 86400 seconds equals. [23:28:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeNVm [23:28:52] [02miraheze/mw-config] 07paladox 03e5c83c0 - Remove EventLogging from ManageWiki, promote to global for CentralNotice [23:28:53] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:28:55] [02mw-config] 07paladox opened pull request 03#2842: Remove EventLogging from ManageWiki, promote to global for CentralNotice - 13https://git.io/JeNVY [23:29:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeNVO [23:29:16] [02miraheze/mw-config] 07paladox 03339d341 - Update LocalSettings.php [23:29:18] [02mw-config] 07paladox synchronize pull request 03#2842: Remove EventLogging from ManageWiki, promote to global for CentralNotice - 13https://git.io/JeNVY [23:29:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeNVs [23:29:47] [02miraheze/mw-config] 07paladox 03f3f6846 - Update LocalExtensions.php [23:29:48] [02mw-config] 07paladox synchronize pull request 03#2842: Remove EventLogging from ManageWiki, promote to global for CentralNotice - 13https://git.io/JeNVY [23:30:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeNVG [23:30:07] [02miraheze/mw-config] 07paladox 03e3747ec - Update GlobalExtensions.php [23:30:09] [02mw-config] 07paladox synchronize pull request 03#2842: Remove EventLogging from ManageWiki, promote to global for CentralNotice - 13https://git.io/JeNVY [23:31:29] miraheze/mw-config/paladox-patch-1/e3747ec - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/630466486 [23:31:51] [02mw-config] 07paladox closed pull request 03#2842: Remove EventLogging from ManageWiki, promote to global for CentralNotice - 13https://git.io/JeNVY [23:31:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JeNVC [23:31:54] [02miraheze/mw-config] 07paladox 032ca6018 - Remove EventLogging from ManageWiki, promote to global for CentralNotice (#2842) * Remove EventLogging from ManageWiki, promote to global for CentralNotice * Update LocalSettings.php * Update LocalExtensions.php * Update GlobalExtensions.php [23:36:31] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [23:36:33] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:53:24] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666]