[00:00:35] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.84, 0.44, 0.17
[00:01:06] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is UNKNOWN: NRPE: Unable to read output
[00:01:07] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is UNKNOWN: NRPE: Unable to read output
[00:01:14] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output
[00:01:18] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Daemon on bacula1 is OK: PROCS OK: 2 processes with UID = 116 (bacula)
[00:01:23] <icinga-miraheze>	 RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 475282 MB (99% inode=99%);
[00:01:27] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output
[00:03:06] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 78529 files, 1.738GB, 2019-02-09 06:33:00 (2.0 weeks ago)
[00:03:09] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Diff, 55597 files, 57.01GB, 2019-02-17 02:09:00 (5.9 days ago)
[00:03:13] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1 files, 0B, 2019-02-10 00:05:00 (1.9 weeks ago)
[00:03:28] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CRITICAL: Diff, 409214 files, 30.47GB, 2019-01-20 05:17:00 (4.8 weeks ago)
[00:15:52] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[00:17:54] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[00:43:52] <ZppixBot>	 Hello UsingDomain! If you have any questions feel free to ask and someone should answer soon.
[00:44:29] <UsingDomain>	 Hello, can I use a free domain like a dot tk to point to my page? Thank you
[00:45:32] <paladox>	 hi, yes
[01:00:30] <Not-8d6a>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[01:00:32] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhFxS
[01:00:33] <Not-8d6a>	 [02miraheze/puppet] 07paladox 036c9b66f - Spread backups across multiple files This spreads db4 so that it's 5x80 (5gb each file) Same for static 5x50 (5gb each file)
[01:00:35] <Not-8d6a>	 [02puppet] 07paladox opened pull request 03#986: Spread backups across multiple files - 13https://git.io/fhFx9
[01:01:53] <Not-8d6a>	 [02puppet] 07paladox closed pull request 03#986: Spread backups across multiple files - 13https://git.io/fhFx9
[01:01:55] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFxH
[01:01:56] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0396225ce - Spread backups across multiple files (#986) This spreads db4 so that it's 5x80 (5gb each file) Same for static 5x50 (5gb each file)
[01:01:58] <Not-8d6a>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[01:01:59] <Not-8d6a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[01:30:44] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:32:43] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[01:34:09] <paladox>	 ah
[01:46:09] <icinga-miraheze>	 RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[01:46:48] <icinga-miraheze>	 RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:47:48] <paladox>	 !log mass restarting of bacula-fd accross all servers
[01:47:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:53:19] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:55:18] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[02:05:41] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFpq
[02:05:42] <Not-8d6a>	 [02miraheze/IncidentReporting] 07JohnFLewis 03f678234 - minor change to outage logic
[02:06:36] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhFpm
[02:06:38] <Not-8d6a>	 [02miraheze/mediawiki] 07JohnFLewis 036719bc5 - update IR
[02:15:07] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: no terminated jobs
[02:19:21] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output
[02:20:56] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFp8
[02:20:57] <Not-8d6a>	 [02miraheze/puppet] 07paladox 038580837 - Put bconsole pass in ""
[02:21:49] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: puppet1-fd
[02:22:18] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFpB
[02:22:19] <Not-8d6a>	 [02miraheze/puppet] 07paladox 038bad574 - Update bacula-fd.conf
[02:22:27] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures
[02:23:18] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFpR
[02:23:20] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03df31d19 - Update bacula-dir.conf
[02:24:27] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[02:28:40] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output
[02:30:41] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CRITICAL: Diff, 409214 files, 30.47GB, 2019-01-20 05:17:00 (4.8 weeks ago)
[02:39:17] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFpz
[02:39:19] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0324c1bd9 - Fix puppet backup location
[02:41:51] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is UNKNOWN: NRPE: Unable to read output
[02:44:20] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: puppet1-fd
[02:44:44] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is UNKNOWN: NRPE: Unable to read output
[02:46:44] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CRITICAL: Diff, 409214 files, 30.47GB, 2019-01-20 05:17:00 (4.8 weeks ago)
[03:02:17] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 2895 files, 4.417MB, 2019-02-23 03:01:00 (1.3 minutes ago)
[03:14:30] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 52195 MB (14% inode=96%);
[03:25:53] <Not-8d6a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhFpj
[03:25:55] <Not-8d6a>	 [02miraheze/mw-config] 07paladox 03330c991 - Compress wiki baobabarchiveswiki
[03:27:46] <paladox>	 !log run compressOld.php on baobabarchiveswiki (on mw1)
[03:27:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[05:03:47] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 391
[05:09:47] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[05:42:21] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:44:16] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 10 May 2019 08:37:39 PM GMT +0000.
[06:03:47] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 515
[06:13:47] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[10:46:30] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40296 MB (10% inode=96%);
[11:59:55] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:01:54] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[13:24:27] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:26:32] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[13:32:57] <Not-8d6a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbJ0
[13:32:58] <Not-8d6a>	 [02miraheze/dns] 07paladox 03b6335b3 - Revert "depool cp5" This reverts commit 80796ca05725870accce53db28be996bd05ca8e7.
[14:39:18] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.24, 0.06, 0.02
[14:40:00] <icinga-miraheze>	 RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 12287 MB (29% inode=98%);
[14:43:50] <icinga-miraheze>	 RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 22921 bytes in 0.011 second response time
[14:44:07] <icinga-miraheze>	 RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 2 processes with command name 'php-fpm7.3'
[14:45:42] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:46:25] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[14:47:04] <Not-8d6a>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[14:47:06] <Not-8d6a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhbUm
[14:47:08] <Not-8d6a>	 [02miraheze/ssl] 07paladox 031f4a555 - Remove fikcyjnatv.pl ssl certificate Domain is not pointing at us && also has not been doing so for 2 days.
[14:47:09] <Not-8d6a>	 [02ssl] 07paladox opened pull request 03#145: Remove fikcyjnatv.pl ssl certificate - 13https://git.io/fhbUY
[14:47:28] <Not-8d6a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/fhbU3
[14:47:30] <Not-8d6a>	 [02miraheze/dns] 07paladox 038a2738a - Remove fikcyjnatv.pl from dns Domain is not pointing at our nameservers.
[14:47:40] <Not-8d6a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/fhbUs
[14:47:42] <Not-8d6a>	 [02miraheze/ssl] 07paladox 037b05a63 - Delete fikcyjnatv.pl.crt
[14:47:43] <Not-8d6a>	 [02ssl] 07paladox synchronize pull request 03#145: Remove fikcyjnatv.pl ssl certificate - 13https://git.io/fhbUY
[14:47:49] <Not-8d6a>	 [02ssl] 07paladox closed pull request 03#145: Remove fikcyjnatv.pl ssl certificate - 13https://git.io/fhbUY
[14:47:51] <Not-8d6a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fhbUG
[14:47:52] <Not-8d6a>	 [02miraheze/ssl] 07paladox 035137f4b - Remove fikcyjnatv.pl ssl certificate (#145) * Remove fikcyjnatv.pl ssl certificate Domain is not pointing at us && also has not been doing so for 2 days. * Delete fikcyjnatv.pl.crt
[14:48:19] <Not-8d6a>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[14:48:20] <Not-8d6a>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[14:50:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbUc
[14:50:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 0304801aa - BOT: Updating services config for wikis
[16:52:19] <Reception123>	 paladox: task for you :) https://phabricator.miraheze.org/T4130
[16:52:35] <paladox>	 heh
[16:52:47] <ZppixBot>	 Title: [ ⚓ T4130 Error in the creation of thumbnails (http://kurwowizja.miraheze.org). ] - phabricator.miraheze.org
[16:55:38] <Not-8d6a>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbkt
[16:55:39] <Not-8d6a>	 [02miraheze/ManageWiki] 07paladox 0371b2ee9 - Fix "Undefined index: extensions"
[16:56:49] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhbkq
[16:56:50] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 032aaa9f0 - Update MW
[16:58:10] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhbkY
[16:58:12] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 031aa89c9 - Update News
[17:29:50] <Reception123>	 paladox: also https://meta.miraheze.org/wiki/Stewards%27_noticeboard#Flow_Discussion_Bar_is_not_working
[17:29:52] <ZppixBot>	 Title: [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org
[17:30:08] <paladox>	 NoParserException?
[17:30:10] <paladox>	 hmm
[17:30:16] <paladox>	 i saw something about flow in the logs
[17:37:57] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhbkd
[17:37:58] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 031f3826c - Update Flow
[17:54:09] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_parsoid]
[17:56:07] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbIT
[17:56:09] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03964da92 - parsoid: Use version 0.10.0 instead of master
[17:58:09] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[20:59:11] <paladox>	 !log upgrading puppet* to 6.3 (puppetserver to 6.2) on puppet1
[20:59:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:02:55] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:04:53] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 10 May 2019 08:37:39 PM GMT +0000.
[21:22:55] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:24:50] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 10 May 2019 08:37:39 PM GMT +0000.
[23:34:58] <icinga-miraheze>	 PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 118 failures. Last run 2 minutes ago with 118 failures. Failed resources (up to 3 shown): File[wiki.kybermc.tk_private],File[drag.lgbt],File[drag.lgbt_private],File[newsnow.wiki]
[23:42:58] <icinga-miraheze>	 RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[23:56:14] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+3/-3/±0] 13https://git.io/fhbOf
[23:56:15] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0395f6d6f - Migrate php_ini.rb, ini.rb and ordered_yaml.rb to modern puppet custom functions