[00:08:50] <ZppixBot>	 Hello alessivs! If you have any questions feel free to ask and someone should answer soon.
[06:26:15] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2950 MB (12% inode=94%);
[07:05:11] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj9AD
[07:05:13] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot 030bb45f0 - BOT: Updating services config for wikis
[08:15:28] <RhinosF1>	 Reception123: db4 disk space critical and file uploads blocked - see https://phabricator.miraheze.org/T4599
[08:15:29] <ZppixBot>	 [ ⚓ T4599 Disk space critical on db4 ] - phabricator.miraheze.org
[08:17:34] <RhinosF1>	 Paladox: ^^ If you're around
[08:18:07] <MH-Discord>	 <RhinosF1> @System Administrators ^
[08:34:07] <RhinosF1>	 PuppyKun, SPF|Cloud: see above UBN!
[08:35:34] <Reception123>	 Will look
[08:36:36] <Reception123>	 RhinosF1: db4 has 19GB.
[08:36:52] <Reception123>	 paladox just moved matomo to db5 so it would not be possible for it to be full already
[08:37:00] <Reception123>	 "critical" is just under a certain amount
[08:37:16] <RhinosF1>	 Reception123: Icinga says critical and since last night file uploads failed so I assumed related
[08:37:36] <Reception123>	 nope, not at all
[08:37:39] <Reception123>	 not sure what to say about the files
[08:38:24] <RhinosF1>	 Reception123: logs say any more information?? Or permissions setup right for the file system
[08:38:39] <Reception123>	 mfs#185.52.1.144:9421  450G  450G  176M 100% /mnt/mediawiki-static
[08:38:42] <Reception123>	 now that is not good
[08:39:19] <RhinosF1>	 That's the issue then.
[08:39:34] <Reception123>	 yeah, no idea why Icinga didn't say anything about space on the filesystem
[08:39:39] <RhinosF1>	 I'll leave updating phab to you Reception123
[08:39:50] <RhinosF1>	 And yeah we should add that.
[08:42:43] <Reception123>	 Well 1) I've got to be somewhere and won't have access soon 2) there's nothing I can do alone regarding the filesystem 3) we probably need a new server which I also can't do without approval
[08:44:13] <RhinosF1>	 Not good. That likely needs Labster or John then to approve. Shall I do a separate task for tracking mediawiki-static in Icinga Reception123
[08:53:52] <Reception123>	 If you want, though I'm not quite sure what the issue is exactly 
[08:54:00] <Reception123>	 Since I'm not too familiar with the new lizardfs
[08:54:32] <RhinosF1>	 Okay
[11:04:18] <Reception123>	 RhinosF1: I might be able to have a look in a bit, though I doubt there's anything I can do
[11:19:03] <Reception123>	 RhinosF1: the thing is it seems like the server has 150GB but only 132 of that is allocated to the file system the rest is not being used
[11:20:23] <Reception123>	 RhinosF1: yup https://github.com/miraheze/puppet/blob/3f4885ee4a2006bd2ac8070b08d74f65c1a6454b/modules/lizardfs/templates/mfschunkserver.cfg.erb#L67
[11:20:23] <ZppixBot>	 [ puppet/mfschunkserver.cfg.erb at 3f4885ee4a2006bd2ac8070b08d74f65c1a6454b · miraheze/puppet · GitHub ] - github.com
[11:21:24] <Not-9fc7>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fj9hv
[11:21:26] <Not-9fc7>	 [02miraheze/puppet] 07Reception123 03da87276 - HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) @paladox @JohnFLewis please approve ASAP (not sure if this will impact anything but won't push without approval)
[11:21:27] <Not-9fc7>	 [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbiAS
[11:21:29] <Not-9fc7>	 [02puppet] 07Reception123 opened pull request 03#1058: HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) - 13https://git.io/fj9hJ
[11:22:36] <Reception123>	 RhinosF1: ^ since I don't know exactly and I wouldn't want to break anything or make things worse I'll wait for paladox or John to approve this before I go through with it
[11:44:09] <icinga-miraheze>	 PROBLEM - lizardfs1 Disk Space on lizardfs1 is WARNING: DISK WARNING - free space: / 16699 MB (10% inode=98%);
[11:47:34] <Not-9fc7>	 [02puppet] 07paladox commented on pull request 03#1058: HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) - 13https://git.io/fj9h8
[11:47:41] <Not-9fc7>	 [02puppet] 07paladox closed pull request 03#1058: HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) - 13https://git.io/fj9hJ
[11:47:43] <Not-9fc7>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj9h4
[11:47:44] <Not-9fc7>	 [02miraheze/puppet] 07Reception123 0352acf93 - HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) (#1058) @paladox @JohnFLewis please approve ASAP (not sure if this will impact anything but won't push without approval)
[11:48:21] <Not-9fc7>	 [02puppet] 07Reception123 commented on pull request 03#1058: HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) - 13https://git.io/fj9hu
[11:48:48] <paladox>	 Reception123: ^
[11:49:04] <Reception123>	 paladox: do you have access?
[11:49:26] <Not-9fc7>	 [02puppet] 07paladox commented on pull request 03#1058: HDD_LEAVE_SPACE_DEFAULT to 12 GB (lack of disk space) - 13https://git.io/fj9hz
[11:49:59] <Reception123>	 paladox: is "sudo service lizardfs-chunkserver restart" safe? (Is there a risk of something going wrong or shutting down)
[11:50:33] <paladox>	 Reception123: not atm, left a comment
[11:50:34] <paladox>	 Yes, it’s the only way to change that config.
[11:51:08] <Reception123>	 ok
[11:51:16] <Reception123>	 !log sudo service lizardfs-chunkserver restart
[11:51:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[11:51:32] <paladox>	 Make sure pupprt has ran reception123
[11:52:19] <Reception123>	 paladox: it would've since you merged at 47 and now it's 52
[11:52:55] <Reception123>	 paladox: also figured we still had dumps on static and am in the process of removing but I'm making a backup just in case someone would still need it
[11:53:10] <paladox>	 Ok
[11:54:13] <Reception123>	 mfs#185.52.1.144:9421  300G  300G  124M 100% /mnt/mediawiki-static
[11:54:16] <Reception123>	 doesn't seem to be working
[11:55:00] <paladox>	 Reception123: I’m pretty sure when you ran that restart command, that the old value was still there
[11:55:01] <paladox>	 Try now?
[11:56:13] <Reception123>	 !log removed /srv/mediawiki-static/dumps (obsolete, replaced by datadump_
[11:56:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[11:56:38] <Reception123>	 !log  sudo service lizardfs-chunkserver restart (again)
[11:56:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[11:57:24] <Reception123>	 paladox: seems to have worked now, though this is just a temporary solution. we need a new server I guess
[11:58:45] <paladox>	 Ok
[12:05:04] <RhinosF1>	 Reception123: per your comment, T4603
[12:06:47] <Reception123>	 RhinosF1: thanks, will make it more specific when I get the details
[12:07:04] <Reception123>	 but we're at 5.2 G right now, so uploads should definitely be back
[12:08:38] <RhinosF1>	 Good
[12:10:05] <Not-9fc7>	 [02puppet] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbiAS
[12:10:06] <Not-9fc7>	 [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-1
[12:14:15] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2646 MB (10% inode=94%);
[16:14:17] <paladox>	 !log sudo service lizardfs-chunkserver restart on lizardfs[123]
[16:14:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:15:46] <RhinosF1>	 JohnLewis: You might want to see https://phabricator.miraheze.org/T4603
[16:15:47] <ZppixBot>	 [ ⚓ T4603 Purchase lizardfs4 (due to lack of space) ] - phabricator.miraheze.org
[17:00:14] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHJ1
[17:00:15] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot 03e16f93a - BOT: Updating services config for wikis
[17:03:15] <JohnLewis>	 RhinosF1: am aware
[17:04:17] <RhinosF1>	 JohnLewis: Good, saw you fastracked approving stuff re recently with Labster inactive
[17:05:23] <JohnLewis>	 I'm cautious over 2 big increases 
[17:06:01] <RhinosF1>	 JohnLewis: okay, I think we have about 5GB for now
[17:07:01] <RhinosF1>	 Plus I think there's 12GB reserved in the file system
[17:08:52] <paladox>	 we have 14gb left
[17:09:38] <paladox>	 also we won't be able to reduce that config anymore after finding out more info on the config. (It stops new chunks, but existing chunks could still fill up)
[17:11:57] <RhinosF1>	 Paladox: okay, I thought the task said 5.2 - where's the rest from?
[17:12:22] <RhinosF1>	 Or I heard 5.2 somewhere
[17:12:25] <paladox>	 it was because, the chunkserver was restarted before the change took affect.
[17:12:39] <paladox>	 you need to restart the chunkserver after puppet has applied the new config
[17:12:43] <RhinosF1>	 Paladox: okay
[17:12:44] <paladox>	 not before
[17:15:00] <JohnLewis>	 well duh
[17:17:09] <Not-9fc7>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHJd
[17:17:11] <Not-9fc7>	 [02miraheze/puppet] 07paladox 0385a038a - matomo: increase memory_limit to 170M
[17:18:39] <Not-9fc7>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHJN
[17:18:41] <Not-9fc7>	 [02miraheze/puppet] 07paladox 0341b481b - Update init.pp
[17:36:59] <Not-9fc7>	 [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/fjHUC
[17:37:01] <Not-9fc7>	 [02miraheze/puppet] 07paladox 0347d4483 - Revert "Update init.pp" This reverts commit 41b481bb3c843283867b46615ecaad90660616d8.
[17:37:02] <Not-9fc7>	 [02miraheze/puppet] 07paladox 03957b3f0 - Revert "matomo: increase memory_limit to 170M" This reverts commit 85a038a944e47978354789b457f9d94147a1f322.
[17:37:04] <Not-9fc7>	 [02miraheze/puppet] 07paladox 03201edc0 - php: Decrease memory_limit for cli to 128M by default
[18:35:01] <ZppixBot>	 Hello Albionic! If you have any questions feel free to ask and someone should answer soon.
[18:39:14] <RhinosF1>	 If I was to write a help page on how to set up an extension and configure from MW should I just put it in Help:Extension name and where can it be advertised? Like adding a link to it from MW/Ext
[19:00:43] <Reception123>	 Yeah that could work 
[19:01:00] <Reception123>	 Though I don't know how we could link since currently links are to mw.org
[19:35:14] <RhinosF1>	 I'll create the page at [[Help: Extension name]] later and will look the other part later.
[19:35:48] <Reception123>	 ok
[19:36:51] <RhinosF1>	 Reception123: check PMs
[20:10:12] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHki
[20:10:14] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot 039e236a6 - BOT: Updating services config for wikis
[20:35:10] <icinga-miraheze>	 PROBLEM - wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Mon 19 Aug 2019 08:32:02 PM GMT +0000).
[20:35:24] <icinga-miraheze>	 PROBLEM - www.wikiescola.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiescola.com.br' expires in 15 day(s) (Mon 19 Aug 2019 08:32:02 PM GMT +0000).
[20:35:25] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHkN
[20:35:26] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 034c91d5d - Bot: Update SSL cert for wikiescola.com.br
[20:35:26] <icinga-miraheze>	 PROBLEM - www.schulwiki.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.schulwiki.de' expires in 15 day(s) (Mon 19 Aug 2019 08:33:22 PM GMT +0000).
[20:35:39] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHkA
[20:35:41] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 03050f70e - Bot: Update SSL cert for www.schulwiki.de
[20:37:29] <icinga-miraheze>	 PROBLEM - www.runeasy.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.runeasy.nl' expires in 15 day(s) (Mon 19 Aug 2019 08:34:11 PM GMT +0000).
[20:37:42] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHkx
[20:37:44] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 03f768a11 - Bot: Update SSL cert for www.runeasy.nl
[20:37:49] <icinga-miraheze>	 PROBLEM - www.openonderwijs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.openonderwijs.org' expires in 15 day(s) (Mon 19 Aug 2019 08:35:00 PM GMT +0000).
[20:38:02] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHkp
[20:38:04] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 038e4bb6d - Bot: Update SSL cert for www.openonderwijs.org
[20:40:03] <icinga-miraheze>	 PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Mon 19 Aug 2019 08:36:55 PM GMT +0000).
[20:40:13] <icinga-miraheze>	 PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 15 day(s) (Mon 19 Aug 2019 08:36:55 PM GMT +0000).
[20:40:27] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIv
[20:40:28] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 03b89d9fb - Bot: Update SSL cert for marinebiodiversitymatrix.org
[20:41:09] <icinga-miraheze>	 PROBLEM - www.evanswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.evanswiki.org' expires in 15 day(s) (Mon 19 Aug 2019 08:37:49 PM GMT +0000).
[20:41:09] <icinga-miraheze>	 PROBLEM - www.eerstelijnszones.be - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eerstelijnszones.be' expires in 15 day(s) (Mon 19 Aug 2019 08:38:52 PM GMT +0000).
[20:41:18] <icinga-miraheze>	 PROBLEM - eerstelijnszones.be - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eerstelijnszones.be' expires in 15 day(s) (Mon 19 Aug 2019 08:38:52 PM GMT +0000).
[20:41:22] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIf
[20:41:24] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 03e9cc6b0 - Bot: Update SSL cert for www.evanswiki.org
[20:43:09] <icinga-miraheze>	 RECOVERY - www.evanswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.evanswiki.org' will expire on Fri 01 Nov 2019 07:41:16 PM GMT +0000.
[20:43:10] <icinga-miraheze>	 RECOVERY - wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Fri 01 Nov 2019 07:35:18 PM GMT +0000.
[20:43:24] <icinga-miraheze>	 RECOVERY - www.wikiescola.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiescola.com.br' will expire on Fri 01 Nov 2019 07:35:18 PM GMT +0000.
[20:43:26] <icinga-miraheze>	 RECOVERY - www.schulwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.schulwiki.de' will expire on Fri 01 Nov 2019 07:35:33 PM GMT +0000.
[20:43:28] <icinga-miraheze>	 RECOVERY - www.runeasy.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.runeasy.nl' will expire on Fri 01 Nov 2019 07:37:35 PM GMT +0000.
[20:43:49] <icinga-miraheze>	 RECOVERY - www.openonderwijs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.openonderwijs.org' will expire on Fri 01 Nov 2019 07:37:56 PM GMT +0000.
[20:44:03] <icinga-miraheze>	 RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Fri 01 Nov 2019 07:40:20 PM GMT +0000.
[20:44:13] <icinga-miraheze>	 RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Fri 01 Nov 2019 07:40:20 PM GMT +0000.
[20:46:15] <icinga-miraheze>	 PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Mon 19 Aug 2019 08:44:08 PM GMT +0000).
[20:46:29] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIU
[20:46:31] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 03d33bd94 - Bot: Update SSL cert for dariawiki.org
[20:47:08] <icinga-miraheze>	 PROBLEM - www.bushcraftpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bushcraftpedia.org' expires in 15 day(s) (Mon 19 Aug 2019 08:45:02 PM GMT +0000).
[20:47:22] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIT
[20:47:23] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 0353b22a6 - Bot: Update SSL cert for www.bushcraftpedia.org
[20:47:42] <icinga-miraheze>	 PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Mon 19 Aug 2019 08:44:08 PM GMT +0000).
[20:48:54] <icinga-miraheze>	 PROBLEM - www.bestpractice.technology - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bestpractice.technology' expires in 15 day(s) (Mon 19 Aug 2019 08:45:55 PM GMT +0000).
[20:49:09] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIk
[20:49:10] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 036a39727 - Bot: Update SSL cert for www.bestpractice.technology
[20:50:46] <icinga-miraheze>	 PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Mon 19 Aug 2019 08:47:53 PM GMT +0000).
[20:51:00] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHII
[20:51:02] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 035a6990e - Bot: Update SSL cert for allthetropes.org
[20:51:29] <icinga-miraheze>	 PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 15 day(s) (Mon 19 Aug 2019 08:47:53 PM GMT +0000).
[20:52:17] <icinga-miraheze>	 PROBLEM - wintermoor.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wintermoor.net' expires in 15 day(s) (Mon 19 Aug 2019 08:48:43 PM GMT +0000).
[20:52:29] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIL
[20:52:31] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 032e229b2 - Bot: Update SSL cert for wintermoor.net
[20:52:46] <icinga-miraheze>	 RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Fri 01 Nov 2019 07:50:54 PM GMT +0000.
[20:52:54] <icinga-miraheze>	 RECOVERY - www.bestpractice.technology - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bestpractice.technology' will expire on Fri 01 Nov 2019 07:49:01 PM GMT +0000.
[20:53:08] <icinga-miraheze>	 RECOVERY - www.bushcraftpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bushcraftpedia.org' will expire on Fri 01 Nov 2019 07:47:15 PM GMT +0000.
[20:53:11] <icinga-miraheze>	 PROBLEM - wikiverte.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiverte.pl' expires in 15 day(s) (Mon 19 Aug 2019 08:49:22 PM GMT +0000).
[20:53:24] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIq
[20:53:26] <Not-9fc7>	 [02miraheze/ssl] 07MirahezeSSLBot 038cc6df8 - Bot: Update SSL cert for wikiverte.pl
[20:53:30] <icinga-miraheze>	 RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Fri 01 Nov 2019 07:50:54 PM GMT +0000.
[20:53:42] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Fri 01 Nov 2019 07:46:23 PM GMT +0000.
[20:54:15] <icinga-miraheze>	 RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Fri 01 Nov 2019 07:46:23 PM GMT +0000.
[21:03:02] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[21:03:30] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.76, 1.18, 0.64
[21:03:38] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 90%
[21:03:57] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb
[21:04:02] <RhinosF1>	 Reception123, JohnLewis: That don't sound good ^
[21:04:17] <icinga-miraheze>	 RECOVERY - wintermoor.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wintermoor.net' will expire on Fri 01 Nov 2019 07:52:23 PM GMT +0000.
[21:04:30] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb
[21:05:11] <icinga-miraheze>	 RECOVERY - wikiverte.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiverte.pl' will expire on Fri 01 Nov 2019 07:53:18 PM GMT +0000.
[21:05:30] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.90, 1.04, 0.65
[21:06:32] <RhinosF1>	 looks like we're still up at least
[21:25:10] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjHIl
[21:25:12] <Not-9fc7>	 [02miraheze/services] 07MirahezeSSLBot 0302d2a4f - BOT: Updating services config for wikis
[21:26:40] <paladox>	 RhinosF1: I’m mobile, but things look like they recovered
[21:27:19] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 28427 MB (7% inode=96%);
[21:27:57] <RhinosF1>	 paladox: see this https://usercontent.irccloud-cdn.com/file/tuXwLZ5z/Screenshot%202019-08-03%20at%2022.27.41.png
[21:29:10] <paladox>	 that's cp2
[21:30:18] <paladox>	 hmm
[21:30:30] <paladox>	 it's still down
[21:30:43] <paladox>	 I'm on the laptop to check quickly.
[21:30:53] <paladox>	 i see:
[21:30:55] <paladox>	 boot.mw1                       probe      Sick                0/5 Sat, 03 Aug 2019 21:00:50 GMT
[21:31:17] <RhinosF1>	 paladox: I guessed the GDNSD/other errors might have been just things moaning cp2 was down
[21:31:28] <paladox>	 It's the dns depooling cp2
[21:31:38] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 25%
[21:32:04] <paladox>	 Well it means the dns has problems with cp2 then depools it.
[21:32:41] <RhinosF1>	 paladox: ah the RECOVERY sounds a bit better
[21:32:54] <paladox>	 it's not recovered
[21:33:32] <RhinosF1>	 paladox: error rate is less, varnish backends (mw[123]) still down
[21:33:39] <paladox>	 yup
[21:33:43] <RhinosF1>	 and GDNSD
[21:33:50] <RhinosF1>	 datacentres
[21:35:38] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 68%
[21:35:59] <RhinosF1>	 paladox: ah, error rate back up again - any idea whats up
[21:36:01] <paladox>	 !log restart varnish and nginx on cp2
[21:36:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:38:09] <icinga-miraheze>	 PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4034 bytes in 0.395 second response time
[21:38:31] <RhinosF1>	 paladox: ^ even worse?
[21:39:16] <paladox>	 Not worse (it's the same) from what i tell mw* is throwing 503 for it
[21:39:32] <RhinosF1>	 paladox: ah
[21:40:14] <paladox>	 until mw* is pooled, it'll be throwing
[21:40:40] <RhinosF1>	 paladox: okay, makes sense to throw and error when it's not pooled tbh
[21:51:51] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 2.00, 1.83, 1.29
[21:53:13] <paladox>	 !log restart stunnel4 on cp2
[21:53:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:53:45] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.19, 1.62, 1.27
[21:54:10] <icinga-miraheze>	 RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1496 bytes in 0.726 second response time
[21:54:30] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[21:55:02] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[21:55:38] <paladox>	 RhinosF1 should be recovering ( SPF|Cloud found the cause )
[21:55:38] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4%
[21:55:57] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:56:50] <RhinosF1>	 paladox: ^ yep! thanks SPF|Cloud
[21:58:54] <SPF|Cloud>	 No problem :P
[23:32:30] <paladox>	 !log deleting old backups on bacula1 (fresh backups will start being generated in ~30mins)
[23:32:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:33:12] <icinga-miraheze>	 RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 470830 MB (98% inode=99%);
[23:33:18] <paladox>	 !log apt-get upgrade on bacula1
[23:33:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:41:05] <Hispano76>	 paladox, would you have time? https://phabricator.miraheze.org/T4606 
[23:41:06] <ZppixBot>	 [ ⚓ T4606 Database name change and ulr for UcroniasWiki ] - phabricator.miraheze.org