[00:02:35] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2061 MB (10% inode=93%);
[00:41:53] <QuIRC>	 Good evening Miraheze
[01:40:24] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfb0o
[01:40:25] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 0366b3e21 - BOT: Updating services config for wikis
[03:45:13] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbzE
[03:45:15] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 03b4b54c6 - BOT: Updating services config for wikis
[04:36:03] <icinga-miraheze>	 PROBLEM - www.mashedpotaytoes.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.mashedpotaytoes.com' expires in 15 day(s) (Mon 06 Jul 2020 04:32:42 GMT +0000).
[04:37:45] <Not-f378>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfbgl
[04:37:46] <Not-f378>	 [02miraheze/ssl] 07MirahezeSSLBot 035153c5e - Bot: Update SSL cert for www.mashedpotaytoes.com
[04:42:09] <icinga-miraheze>	 RECOVERY - www.mashedpotaytoes.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mashedpotaytoes.com' will expire on Fri 18 Sep 2020 03:37:39 GMT +0000.
[04:54:47] <hd1>	 that can (and should) be put in a cronjob
[05:12:32] <Not-f378>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://git.io/Jfb2W
[05:12:34] <Not-f378>	 [02miraheze/dns] 07Reception123 03f95664a - add hololive.wiki
[05:12:35] <Not-f378>	 [02dns] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbQXl
[05:12:40] <Not-f378>	 [02dns] 07Reception123 opened pull request 03#150: add hololive.wiki zone - 13https://git.io/Jfb2l
[05:17:11] <hd1>	 there's also no harm in renewing daily,as the Letsencrypt system doesn't renew unitl you're close to expiration, indeed this was (is?) their recommendation
[05:27:22] <Not-f378>	 [02dns] 07Reception123 reviewed pull request 03#150 commit - 13https://git.io/Jfb26
[05:30:52] <Reception123>	 hd1: well we have renews done by a bot
[05:31:20] <Reception123>	 and I think it should be rather close to expiration?
[05:31:46] <hd1>	 it doesn't matter, LE doesn't renew unless you're close to expiration
[05:32:06] <hd1>	 I have my servers trigger LE every 12 hours, for instance
[05:32:18] <hd1>	 on a cronjob
[06:00:20] <Reception123>	 ah, well for us it's a bit complicated because we've got our certs on GitHub so we need to generate them and then transfer and commit them 
[06:02:05] <hd1>	 Reception123: 
[06:02:07] <hd1>	 ok
[06:41:52] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is CRITICAL: DISK CRITICAL - free space: / 695 MB (3% inode=93%);
[06:49:52] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1418 MB (7% inode=93%);
[06:52:00] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is CRITICAL: DISK CRITICAL - free space: / 543 MB (2% inode=93%);
[06:52:25] <RhinosF1>	 Reception123: can you purge some logs or something ^
[06:53:59] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.97, 1.98, 1.29
[06:55:57] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.80, 1.54, 1.21
[06:57:32] <Reception123>	 I've got no idea but I'll look
[06:59:53] <Reception123>	 RhinosF1: I'm not sure what I can (or should) purge
[07:00:17] <Reception123>	 access logs are 6.3G but I don't get where the rest of the space usage is
[07:00:41] <Reception123>	 ah, cache_storage.bin
[07:02:00] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1405 MB (7% inode=93%);
[07:02:29] <Reception123>	 oh great it fixed itself
[07:02:51] <RhinosF1>	 Reception123: it's flapping. You could try compress the access logs
[07:03:17] <RhinosF1>	 That should be safe and non destructive
[07:03:42] <Reception123>	 RhinosF1: most already are
[07:04:00] <RhinosF1>	 Reception123: hmmm
[07:04:04] <Reception123>	 RhinosF1: I think I can do one though
[07:04:09] <RhinosF1>	 Try that
[07:04:53] <Reception123>	 https://www.irccloud.com/pastebin/7N69JTZ9/
[07:04:54] <ZppixBot>	 [ Snippet | IRCCloud ] - www.irccloud.com
[07:04:56] <Reception123>	 ^ RhinosF1 hmm
[07:05:00] <Reception123>	 that's strange
[07:05:42] <RhinosF1>	 Reception123: that there's a file and zip
[07:06:10] <RhinosF1>	 ?
[07:06:27] <Reception123>	 RhinosF1: yeah, though why is there a zip and the original file
[07:06:45] <RhinosF1>	 Reception123: not deleted properly
[07:07:05] <RhinosF1>	 If they look the same, just delete the old one
[07:07:25] <Reception123>	 RhinosF1: well can't quite tell if they look the same, as unzipping the gz isn't possible either
[07:08:34] <RhinosF1>	 Reception123: maybe just gzip the other one then and wait for paladox
[07:09:40] <Reception123>	 RhinosF1: yeah, gzip with another name I guess
[07:10:32] <RhinosF1>	 Yep
[07:13:41] <Reception123>	 !log gzip access.log.3 on cp8 (lack of disk space)
[07:13:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[07:13:58] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2981 MB (15% inode=93%);
[07:25:10] <RhinosF1>	 paladox: see if there's anything else
[07:25:43] <RhinosF1>	 Reception123: looks to have saved us ~1.5GB
[07:37:41] <Reception123>	 yup
[08:52:43] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.36, 16.85, 14.37
[08:54:40] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 9.50, 13.86, 13.56
[11:23:26] <icinga-miraheze>	 PROBLEM - aman.info.tm - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:01:18] <Sygnal>	 Unexpected error (list index out of range) from catbeard at 2020-06-20 12:01:18.112324. Message was: nope
[13:50:34] * RhinosF1 waves
[13:52:43] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbMJ
[13:52:45] <Not-f378>	 [02miraheze/puppet] 07paladox 03e1cd5fa - cp8: Lower cache size to 8g cp8 disk is almost full so give it an extra gb.
[13:56:24] <JohnLewis>	 Is that necessary? Was SPF consulted?
[13:56:39] <JohnLewis>	 Disk space seems fine on cp8 to me at 87% usage
[13:56:45] <paladox>	 JohnLewis i saw earlier that icinga alerted that we had 500MB
[13:56:49] <paladox>	 before Reception123 cleaned it
[13:56:54] <paladox>	 *left
[13:56:58] <JohnLewis>	 Theres 2.6G on it
[13:57:03] <JohnLewis>	 Your change would make it 3.6G
[13:57:03] <RhinosF1>	 One of the access logs was gzip'd
[13:57:33] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbML
[13:57:34] <Not-f378>	 [02miraheze/puppet] 07paladox 036fc8538 - Revert "cp8: Lower cache size to 8g" This reverts commit e1cd5fa414c6ae784f548916d1582b96df315172.
[13:57:35] <RhinosF1>	 see 07:52 UK time
[13:58:07] <JohnLewis>	 I've removed that access log now
[13:58:10] <JohnLewis>	 It should have been deleted
[13:58:56] <RhinosF1>	 JohnLewis: there was an ungzip and gzip'd one. Reception gzip'd the one that wasn't but wasn't sure if there was a reason for 2 with same name so didnt delete it
[13:59:37] <JohnLewis>	 I used it yesterday and was going to look at it again but didn't
[13:59:58] <RhinosF1>	 Ah
[14:00:11] <RhinosF1>	 Problem solved
[14:00:13] <RhinosF1>	 Reception123: ^
[14:00:38] <Reception123>	 Ah ok
[14:02:18] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.00, 19.12, 15.07
[14:02:37] <RhinosF1>	 paladox: ^ same as yesterday
[14:04:16] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.92, 17.78, 15.08
[14:04:25] * RhinosF1 looks at the ssl alert
[14:05:09] <paladox>	 Nothing much we can do, we're overusing it.
[14:05:39] <RhinosF1>	 Okay
[14:05:59] <RhinosF1>	 The site with the ssl alert seems completely down
[14:08:27] <RhinosF1>	 paladox: is it worth just downtiming the ssl alert for a day and seeing if it comes back up?
[14:08:59] <paladox>	 i suppose so
[14:12:06] <RhinosF1>	 .in 24hours and Paladox: check the sslhost alert from yesterday and deal with
[14:12:06] <ZppixBot>	 RhinosF1: Okay, will remind at 2020-06-21 - 15:12:06BST
[14:15:07] <Not-f378>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfbMh
[14:15:08] <Not-f378>	 [02miraheze/mw-config] 07paladox 0384eb194 - Remove wgJobQueueAggregator - unused
[14:15:10] <Not-f378>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[14:15:11] <Not-f378>	 [02mw-config] 07paladox opened pull request 03#3119: Remove wgJobQueueAggregator - unused - 13https://git.io/JfbMj
[14:15:32] <paladox>	 RhinosF1 you don't really need to set a reminder for that :)
[14:15:43] <paladox>	 i think icinga would notify once the downtime expires *i think*
[14:15:52] * RhinosF1 not sure and will forget
[14:16:10] <RhinosF1>	 JohnLewis: can you also look at something on testwiki? It smells off
[14:16:46] <travis-ci>	 miraheze/mw-config/paladox-patch-2/84eb194 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/700364857
[14:18:21] <JohnLewis>	 sure
[14:19:23] <JohnLewis>	 RhinosF1: what's off?
[14:20:36] <Avengium>	 Hi
[14:20:49] <Avengium>	 i'm from tuscriaturas.miraheze
[14:22:03] <Avengium>	 RhinosF1 , are you busy? someone knows how can i delete completely the page of a file once i don't want that file (or the revisions) any more?
[14:24:52] <RhinosF1>	 You can't delete onwiki revision from the db nicely
[14:26:23] <Avengium>	 like, for example this page that don't exist. Another admin moved the image: https://tuscriaturas.miraheze.org/wiki/Archivo:Logo_Secretos.png
[14:26:24] <ZppixBot>	 [ Archivo:Logo Secretos.png - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org
[14:27:26] <hispano76>	 Hola
[14:27:36] <Avengium>	 Hola hispano76
[14:28:12] <Avengium>	 tengo una duda con el registro de imagenes, y las paginas que se mueven, o cuando se sube la imagen en otra dirección
[14:28:32] <Avengium>	 otro admin de mi comunidad tiene esta imagen: https://tuscriaturas.miraheze.org/wiki/Archivo:Logo_Secreto.png
[14:28:33] <ZppixBot>	 [ Archivo:Logo Secreto.png - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org
[14:28:56] <Avengium>	 la diferencia es Archivo:Logo_Secreto.png vs Archivo:Logo_Secretos.png (con una s)
[14:29:28] <hispano76>	 Avengium:  A ver ¿Dices que no quieres que aparezca el registro de borrado/traslado/etc. en ese archivo con la "s"? 
[14:29:44] <Avengium>	 la versión anterior (con s de plural) está borrada, pero en la nueva sigue marcando que hay un duplicado. ¿eso da algún problema?
[14:30:17] <hispano76>	 Ah, reviso
[14:32:24] <Avengium>	 bien. te espero
[14:32:51] <icinga-miraheze>	 PROBLEM - en.famepedia.co.in - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:33:22] <RhinosF1>	 ^ not resolved, looking
[14:33:38] <hispano76>	 Bien, puede deberse por varios motivos. El sistema/cache no se ha actualizado y tendrías que esperar hasta por unos días hasta que desaparezca o podría ser algún error que desconozco (es más probable que sea el primero). 
[14:33:42] <hispano76>	 Avengium: 
[14:34:01] <Avengium>	 de acuerdo
[14:34:13] <Avengium>	 pero tal como está no da ningún problema. ¿no?
[14:34:23] <hispano76>	 no debería 
[14:34:32] <Avengium>	 de acuerdo. pues muchas gracias
[14:34:41] <hispano76>	 Vale, de nada :)
[14:34:44] <Avengium>	 te mencionaré otra vez cuando tenga dudas
[14:34:55] <icinga-miraheze>	 RECOVERY - en.famepedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.co.in' will expire on Fri 14 Aug 2020 18:32:24 GMT +0000.
[14:34:56] <hispano76>	 De acuerdo
[14:37:46] <JohnLewis>	 !log reduce test2's vCPU to 2 (from 4)
[14:37:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:38:03] <RhinosF1>	 icinga-miraheze: that site is not ok. It still won't resolve
[14:38:50] * RhinosF1 keeps an eye
[14:47:33] <Not-f378>	 [02ssl] 07RhinosF1 opened pull request 03#325: - en.famepedia.co.in - 13https://git.io/JfbDH
[14:48:14] <Not-f378>	 [02ssl] 07RhinosF1 synchronize pull request 03#325: - en.famepedia.co.in - 13https://git.io/JfbDH
[14:49:01] <Not-f378>	 [02ssl] 07RhinosF1 commented on pull request 03#325: - en.famepedia.co.in - 13https://git.io/JfbD7
[14:49:24] <Not-f378>	 [02ssl] 07RhinosF1 edited pull request 03#325: [DNM] - en.famepedia.co.in - 13https://git.io/JfbDH
[14:53:15] <RhinosF1>	 https://phabricator.miraheze.org/T5605#112883
[14:53:16] <ZppixBot>	 [ ⚓ T5605 I want Lets Encrypt certificate ] - phabricator.miraheze.org
[15:04:07] <RhinosF1>	 Ha!
[15:05:36] <RhinosF1>	 ^ that's expected. It's host based
[15:08:34] <RhinosF1>	 told zppix they can join
[15:41:19] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 9 minutes ago with 0 failures
[15:43:57] <icinga-miraheze>	 PROBLEM - baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Mon 06 Jul 2020 15:41:21 GMT +0000).
[15:44:18] <icinga-miraheze>	 PROBLEM - www.baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Mon 06 Jul 2020 15:41:21 GMT +0000).
[15:47:05] <Not-f378>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbS7
[15:47:06] <Not-f378>	 [02miraheze/ssl] 07MirahezeSSLBot 03db2b8e6 - Bot: Update SSL cert for www.baharna.org
[15:51:56] <icinga-miraheze>	 RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Fri 18 Sep 2020 14:46:58 GMT +0000.
[15:52:17] <icinga-miraheze>	 RECOVERY - www.baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Fri 18 Sep 2020 14:46:58 GMT +0000.
[15:57:19] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr1
[15:57:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:57:23] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 51 seconds ago with 1 failures. Failed resources (up to 3 shown): Service[gdnsd]
[15:58:55] <Not-f378>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/Jfb9B
[15:58:57] <Not-f378>	 [02miraheze/ssl] 07Reception123 031b657d1 - rm aman.info.tm.crt (replaced with another domain)
[16:05:02] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:05:10] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:05:15] <icinga-miraheze>	 PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:05:53] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:06:04] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:06:10] <RhinosF1>	 Reception123: ^
[16:06:12] <icinga-miraheze>	 PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:06:33] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:06:45] <Reception123>	 ack, fixed
[16:07:07] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:07:21] <RhinosF1>	 ty
[16:07:22] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[aman.info.tm]
[16:07:45] <Reception123>	 RhinosF1: JohnLewis survey is now up
[16:08:01] <RhinosF1>	 Reception123: ack
[16:08:37] <JohnLewis>	 Reception123: +1
[16:15:46] <Not-f378>	 [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfb9b
[16:15:47] <Not-f378>	 [02miraheze/dns] 07JohnFLewis 03aff4425 - deploy cp6+7 side by side + move from cp* to region DC names
[16:15:49] <JohnLewis>	 paladox: ^
[16:16:14] <Not-f378>	 [02dns] 07Reception123 closed pull request 03#150: add hololive.wiki zone - 13https://git.io/Jfb2l
[16:16:15] <Not-f378>	 [02miraheze/dns] 07Reception123 pushed 032 commits to 03master [+2/-0/±0] 13https://git.io/Jfb9N
[16:16:17] <Not-f378>	 [02miraheze/dns] 07Reception123 037267ceb - Merge pull request #150 from miraheze/Reception123-patch-1 add hololive.wiki zone
[16:23:16] <hispano76>	 Reception123: ping
[16:33:03] <paladox>	 JohnLewis: wow
[16:33:24] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:33:29] <Reception123>	 hispano76: hi
[16:38:25] <hispano76>	 Reception123: Es sobre la encuesta, porque no puedo traduccir el banner? se olvidó o se quiso hacer así? perdona
[16:38:43] <hispano76>	 O mejor dicho, no me aparece el enlace de traducir jeje
[16:39:05] <Reception123>	 hispano76: No estoy seguro de como hacerlo
[16:39:14] <Reception123>	 paladox: could you please make the sitenotice translatable? ^^
[16:39:31] <hispano76>	 Ah, ok. Gracias Reception123 tenía esa duda jeje :)
[16:42:01] <paladox>	 I’m currently mobile re
[16:42:02] <paladox>	 Reception123
[16:42:03] <paladox>	 Voidwalker ^
[16:42:05] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.41, 1.80, 1.17
[16:44:06] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.67, 1.40, 1.11
[16:49:42] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[16:50:00] <RhinosF1>	 Uh oh
[16:50:01] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 61%, RTA = 387.20 ms
[16:50:10] <RhinosF1>	 JohnLewis: ^
[16:50:12] <RhinosF1>	 !s
[16:50:14] <SigmaBot>	 Please wait while I check the status of Miraheze Services.
[16:50:15] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:50:15] <SigmaBot>	 RhinosF1: Status report finished. There are currently 0 dead services and 6 alive services. To view the full report, say !status.
[16:50:48] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[16:51:28] <RhinosF1>	 Reception123: cp3 does not sound happy
[16:52:01] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 5%
[16:52:15] <Reception123>	 nope
[16:52:18] <icinga-miraheze>	 RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 242.32 ms
[16:52:28] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 1.002 second response time
[16:52:53] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[16:55:17] <RhinosF1>	 It's back
[16:55:30] <RhinosF1>	 Reception123: we should watch that
[16:56:45] <Reception123>	 hmm yeah
[18:03:04] <joaquinito01>	 Hello, RhinosF1, i can use Miraheze Toolforge???
[19:07:10] <Reception123>	 !log deleted an entry from bot_passwords
[19:07:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:08:15] <RhinosF1>	 paladox: i'll undo the gban
[19:08:22] <paladox>	 thanks
[19:08:27] <RhinosF1>	 Oh sigma is gone
[19:08:33] <paladox>	 yeh
[19:08:35] <RhinosF1>	 dont matter anyway
[19:13:33] <Reception123>	 !log manually reset password for a user
[19:13:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:18:52] <Not-f378>	 [02miraheze/MirahezeMagic] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfbbO
[19:18:54] <Not-f378>	 [02miraheze/MirahezeMagic] 07JohnFLewis 03886668e - Fix duplicate i18n message
[19:19:20] <Not-f378>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jfbb3
[19:19:22] <Not-f378>	 [02miraheze/mediawiki] 07JohnFLewis 03d5976fb - MM
[19:45:06] <icinga-miraheze>	 PROBLEM - wiki.edhel.online - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.edhel.online and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[19:47:08] <icinga-miraheze>	 RECOVERY - wiki.edhel.online - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.edhel.online' will expire on Fri 18 Sep 2020 06:26:09 GMT +0000.
[20:17:55] <icinga-miraheze>	 PROBLEM - wiki.edhel.online - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.edhel.online and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[20:23:53] <icinga-miraheze>	 RECOVERY - wiki.edhel.online - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.edhel.online' will expire on Fri 18 Sep 2020 06:26:09 GMT +0000.
[21:57:02] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2113 MB (10% inode=93%);
[23:30:55] <Not-f378>	 [02miraheze/dns] 07paladox deleted branch 03Reception123-patch-1
[23:30:57] <Not-f378>	 [02dns] 07paladox deleted branch 03Reception123-patch-1 - 13https://git.io/vbQXl
[23:31:40] <Not-f378>	 [02puppet] 07paladox closed pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfPXL
[23:31:42] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JfNe6
[23:31:43] <Not-f378>	 [02miraheze/puppet] 07Amanda-Catherine 03ddd4288 - Add Reddit and Dropbox to CSP whitelist T5685
[23:31:45] <Not-f378>	 [02miraheze/puppet] 07paladox 039001cc2 - Merge pull request #1388 from Amanda-Catherine/patch-1 Add Reddit and Dropbox to CSP whitelist T5685
[23:31:46] <Not-f378>	 [02puppet] 07paladox commented on pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfNei
[23:36:28] <Not-f378>	 [02puppet] 07paladox closed pull request 03#1401: db8: Increase innodb_buffer_pool_instances to 3 - 13https://git.io/JfyaT
[23:36:31] <Not-f378>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6
[23:36:32] <Not-f378>	 [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS