[00:08:44] <icinga-miraheze>	 PROBLEM - Bacula - Private Git on bacula1 is WARNING: WARNING: Full, 1350 files, 1.432MB, 2018-06-24 00:05:00 (1.1 weeks ago)  
[00:13:31] <paladox>	 ^^ that’s because bacula crashed on Sunday morning as it started its backups
[00:18:56] <JohnLewis>	 yep
[00:24:57] <NeoBogan>	 you need more storage
[00:25:16] <paladox>	 NeoBogan heh?
[00:26:15] <NeoBogan>	 if you’re running out of drivespace then there’s really only two causes: too much crap on existing storage, or not enough storage
[00:26:23] <paladox>	 heh
[00:26:31] <paladox>	 NeoBogan we are increasing our file hosting storage
[00:26:53] <NeoBogan>	 if we nuked paladox’ home-made selfie pr0n from the servers, that’d surely open up 75% of existing capacity for the real miraheze users to utilize
[00:27:07] <paladox>	 lol what
[00:29:33] <JohnLewis>	 disk is irrelevant here
[00:59:04] <icinga-miraheze>	 PROBLEM - Current Load on nfs1 is CRITICAL: CRITICAL - load average: 6.07, 3.03, 1.27  
[01:03:04] <icinga-miraheze>	 RECOVERY - Current Load on nfs1 is OK: OK - load average: 1.17, 2.61, 1.57  
[01:07:05] <icinga-miraheze>	 PROBLEM - Current Load on nfs1 is CRITICAL: CRITICAL - load average: 4.83, 4.01, 2.38  
[01:09:05] <icinga-miraheze>	 RECOVERY - Current Load on mw1 is OK: OK - load average: 3.83, 5.51, 3.97  
[01:36:04] <icinga-miraheze>	 PROBLEM - Bacula - Databases - db4 on bacula1 is WARNING: WARNING: Diff, 45541 files, 52.58GB, 2018-06-17 01:33:00 (2.1 weeks ago)  
[01:41:33] <NeoBogan>	 meet
[01:41:35] <NeoBogan>	 the
[01:41:38] <NeoBogan>	 feebles
[01:41:39] <NeoBogan>	 :)
[01:44:55] <icinga-miraheze>	 PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:46:45] <icinga-miraheze>	 RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate wiki.teessidehackspace.org.uk will expire on Wed 08 Aug 2018 11:02:13 AM GMT +0000.  
[01:49:33] <icinga-miraheze>	 PROBLEM - www.reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:49:43] <icinga-miraheze>	 PROBLEM - wikibase.revi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:49:45] <icinga-miraheze>	 PROBLEM - wiki.campaign-labour.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:49:53] <icinga-miraheze>	 PROBLEM - wiki.nvda-nl.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:50:07] <icinga-miraheze>	 PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:50:15] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:54:03] <icinga-miraheze>	 RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate public.revi.wiki will expire on Fri 31 Aug 2018 09:28:02 PM GMT +0000.  
[01:54:06] <icinga-miraheze>	 RECOVERY - reviwiki.info - PositiveSSLDV on sslhost is OK: OK - Certificate reviwiki.info will expire on Wed 03 Feb 2021 11:59:59 PM GMT +0000.  
[01:54:53] <icinga-miraheze>	 PROBLEM - private.revi.wiki - Comodo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:54:55] <icinga-miraheze>	 PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:59:34] <icinga-miraheze>	 PROBLEM - Bacula - Static on bacula1 is WARNING: WARNING: Diff, 51462 files, 4.306GB, 2018-06-17 01:57:00 (2.1 weeks ago)  
[01:59:36] <icinga-miraheze>	 PROBLEM - wiki.itspugle.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:00:16] <icinga-miraheze>	 PROBLEM - reviwiki.info - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:00:24] <icinga-miraheze>	 PROBLEM - Bacula - Phabricator Static on bacula1 is WARNING: WARNING: Diff, 8269 files, 51.53MB, 2018-06-17 01:58:00 (2.1 weeks ago)  
[02:00:34] <icinga-miraheze>	 PROBLEM - disabled.life - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:01:24] <icinga-miraheze>	 RECOVERY - wiki.itspugle.ga - LetsEncrypt on sslhost is OK: OK - Certificate wiki.itspugle.ga will expire on Wed 12 Sep 2018 07:21:28 AM GMT +0000.  
[02:01:34] <icinga-miraheze>	 RECOVERY - wikibase.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate wikibase.revi.wiki will expire on Mon 06 Aug 2018 01:01:05 AM GMT +0000.  
[02:01:36] <icinga-miraheze>	 RECOVERY - wiki.campaign-labour.org - LetsEncrypt on sslhost is OK: OK - Certificate wiki.campaign-labour.org will expire on Sun 02 Sep 2018 09:32:28 PM GMT +0000.  
[02:01:44] <icinga-miraheze>	 RECOVERY - wiki.nvda-nl.org - LetsEncrypt on sslhost is OK: OK - Certificate wiki.nvda-nl.org will expire on Tue 14 Aug 2018 06:29:16 PM GMT +0000.  
[02:02:04] <icinga-miraheze>	 PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:03:04] <icinga-miraheze>	 PROBLEM - wiki.ldmsys.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:04:14] <icinga-miraheze>	 PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:05:14] <icinga-miraheze>	 PROBLEM - wiki.lostminecraftminers.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:06:24] <icinga-miraheze>	 RECOVERY - disabled.life - LetsEncrypt on sslhost is OK: OK - Certificate disabled.life will expire on Wed 08 Aug 2018 10:31:28 AM GMT +0000.  
[02:08:04] <icinga-miraheze>	 RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate wiki.mikrodev.com will expire on Tue 21 Aug 2018 06:56:14 PM GMT +0000.  
[02:08:54] <icinga-miraheze>	 RECOVERY - wiki.ldmsys.net - LetsEncrypt on sslhost is OK: OK - Certificate wiki.ldmsys.net will expire on Fri 21 Sep 2018 03:48:44 PM GMT +0000.  
[02:09:04] <icinga-miraheze>	 RECOVERY - wiki.lostminecraftminers.org - LetsEncrypt on sslhost is OK: OK - Certificate wiki.lostminecraftminers.org will expire on Wed 08 Aug 2018 10:58:08 AM GMT +0000.  
[02:12:35] <icinga-miraheze>	 PROBLEM - Bacula - Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[02:18:25] <icinga-miraheze>	 PROBLEM - Bacula - Phabricator Static on bacula1 is WARNING: WARNING: Diff, 8269 files, 51.53MB, 2018-06-17 01:58:00 (2.1 weeks ago)  
[02:18:43] <icinga-miraheze>	 PROBLEM - Bacula - Private Git on bacula1 is WARNING: WARNING: Full, 1350 files, 1.432MB, 2018-06-24 00:05:00 (1.2 weeks ago)  
[02:19:33] <icinga-miraheze>	 PROBLEM - Bacula - Static on bacula1 is WARNING: WARNING: Diff, 51462 files, 4.306GB, 2018-06-17 01:57:00 (2.1 weeks ago)  
[02:20:05] <icinga-miraheze>	 PROBLEM - Bacula - Databases - db4 on bacula1 is WARNING: WARNING: Diff, 45541 files, 52.58GB, 2018-06-17 01:33:00 (2.1 weeks ago)  
[03:42:12] <Not-683e>	 [02mw-config] 07The-Voidwalker opened pull request 03#2279: Revert "olegcinemawiki config fixes T3263" - 13https://git.io/figPB
[03:43:01] <Voidwalker>	 can someone merge that ASAP? the wiki is currently non-functional
[03:43:38] <Voidwalker>	 although, it would probably be easier just to try and debug if possible
[03:43:49] <Voidwalker>	 *better not easier
[04:40:53] <Not-683e>	 [02miraheze/mw-config] 07NDKilla pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fixYM
[04:40:55] <Not-683e>	 [02miraheze/mw-config] 07NDKilla 03847eb91 - Fix mwexceptions on olegcinemawiki MWException from line 209 of /srv/mediawiki/w/includes/Autopromote.php: Unrecognized condition Array for autopromotion!
[04:45:24] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[05:01:23] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures  
[06:03:34] <Not-683e>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fP1cn
[06:03:36] <Not-683e>	 [02miraheze/CreateWiki] 07translatewiki 03acb7f41 - Localisation updates from https://translatewiki.net.
[06:42:04] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5412  
[06:55:10] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php dumpBackup.php --wiki atlantisboardwiki --full > /home/reception/atlantisboard02072018.xml
[06:55:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[08:38:04] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[09:16:04] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5417  
[09:44:17] <Not-683e>	 [02mw-config] 07MacFan4000 closed pull request 03#2279: Revert "olegcinemawiki config fixes T3263" - 13https://git.io/figPB
[09:44:34] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.76, 3.41, 2.72  
[09:52:35] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.68, 3.35, 2.99  
[09:56:35] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.22, 3.42, 3.10  
[11:12:03] <icinga-miraheze>	 RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs  
[11:34:35] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.34, 3.26, 2.19  
[11:38:33] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.69, 3.36, 2.51  
[12:06:35] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.38, 3.41, 2.68  
[12:54:34] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.90, 3.35, 2.69  
[12:56:34] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 1.91, 2.81, 2.57  
[13:00:00] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fDmCW
[13:00:02] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 030f5154a - Revert "Fix mwexceptions on olegcinemawiki" This reverts commit 847eb91195e2d76b531f9ca7530cc41f535a1c08.
[13:00:03] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 03e9782fe - Syntax fix
[13:21:35] <icinga-miraheze>	 PROBLEM - Disk Space on swift1 is WARNING: DISK WARNING - free space: / 31886 MB (20% inode=96%):  
[13:22:01] <paladox>	 oh yay
[13:22:06] <paladox>	 31gb
[13:59:34] <icinga-miraheze>	 PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.07, 3.10, 1.90  
[14:01:34] <icinga-miraheze>	 RECOVERY - Current Load on swift1 is OK: OK - load average: 2.89, 2.98, 2.00  
[14:26:13] <paladox>	 !log repool mw1 after swift upload
[14:26:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:22:35] <icinga-miraheze>	 PROBLEM - Disk Space on db4 is WARNING: DISK WARNING - free space: / 76924 MB (20% inode=95%):  
[15:24:41] <paladox>	 !log PURGE BINARY LOGS BEFORE DATE(NOW() - INTERVAL 1 DAY) + INTERVAL 0 SECOND; on db4
[15:24:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:25:05] <icinga-miraheze>	 RECOVERY - Disk Space on db4 is OK: DISK OK - free space: / 92276 MB (25% inode=95%):  
[15:26:07] <JohnLewis>	 Was it really too hard to write the date?
[15:27:38] <paladox>	 JohnLewis what data?
[15:27:46] <paladox>	 oh
[15:28:01] <paladox>	 JohnLewis i doin't know the syntax of by hart
[15:37:04] <MacFan4000>	 paladox: how will we clear up space on swift1?
[15:38:07] <paladox>	 MacFan4000 you have to delete the objects, but we won't be able to cleanup space as we will be adding nfs1
[15:38:13] <paladox>	 once i make sure things are ok
[15:38:17] <paladox>	 and do one final sync
[15:38:43] <MacFan4000>	 Ok
[18:53:57] <khalella>	 why did this one get wacked? i don't see any details https://www.facebook.com/AmandaCatherine123/posts/134671023920255:0 
[18:53:57] <ZppixBot>	 Title: [ I was once highly satisfied with the... - Amanda Catherine | Facebook ] - www.facebook.com
[18:54:39] <Reception123>	 khalella: They were a long term abuser and a vandal
[18:54:51] <Reception123>	 The community voted twice to have them globally locked
[18:54:57] <khalella>	 ha. cool
[18:54:58] <Reception123>	 Their behavior was unacceptable 
[18:55:10] <khalella>	 so they vandalized wikis
[18:55:16] <Reception123>	 yes
[20:15:55] <Not-683e>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fSsx0
[20:15:57] <Not-683e>	 [02miraheze/mw-config] 07paladox 03866796d - Disable uploads for one last sync with swift
[20:27:13] <icinga-miraheze>	 PROBLEM - Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]  
[20:27:24] <paladox>	 !log depool mw1 for one final swift sync
[20:27:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:52:33] <ZppixBot>	 Hello svet! If you have any questions feel free to ask and someone should answer soon.
[21:03:04] <icinga-miraheze>	 PROBLEM - Current Load on nfs1 is CRITICAL: CRITICAL - load average: 4.93, 2.91, 1.33  
[21:05:04] <icinga-miraheze>	 PROBLEM - Current Load on nfs1 is WARNING: WARNING - load average: 3.97, 2.84, 1.47  
[21:07:04] <icinga-miraheze>	 RECOVERY - Current Load on nfs1 is OK: OK - load average: 1.89, 2.74, 1.62  
[21:33:24] <icinga-miraheze>	 PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt]  
[21:41:24] <icinga-miraheze>	 RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures  
[22:23:35] <icinga-miraheze>	 PROBLEM - Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: swift, last run 27 seconds ago with 0 failures  
[22:26:14] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fSGLF
[22:26:15] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 0366f8088 - Update Sitenotice.php
[22:35:14] <icinga-miraheze>	 PROBLEM - HTTP on mw1 is CRITICAL: connect to address 185.52.1.75 and port 80: Connection refused  
[22:35:16] <icinga-miraheze>	 PROBLEM - HTTPS on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused  
[22:35:24] <icinga-miraheze>	 PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refused  
[22:36:04] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw1  
[22:36:14] <icinga-miraheze>	 PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw1  
[22:42:04] <icinga-miraheze>	 PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2627  
[22:45:14] <icinga-miraheze>	 RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures  
[22:57:17] <Not-683e>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fSGqS
[22:57:18] <Not-683e>	 [02miraheze/puppet] 07paladox 03ff81524 - Set client_max_body_size to 0 This is where we store files so we expect large uploads
[23:15:04] <Not-683e>	 [02mw-config] 07Wiki1776 opened pull request 03#2280: enable configurations on unidoswiki - 13https://git.io/fSGmp
[23:15:45] <Not-683e>	 [02mw-config] 07MacFan4000 closed pull request 03#2280: enable configurations on unidoswiki - 13https://git.io/fSGmp
[23:15:47] <Not-683e>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fSGmh
[23:15:48] <Not-683e>	 [02miraheze/mw-config] 07Wiki1776 03f4a2ead - Update LocalSettings.php
[23:16:46] <Not-683e>	 [02services] 07Wiki1776 opened pull request 03#142: add unidoswiki on services - 13https://git.io/fSGYf
[23:16:59] <Not-683e>	 [02services] 07MacFan4000 closed pull request 03#142: add unidoswiki on services - 13https://git.io/fSGYf
[23:17:00] <Not-683e>	 [02miraheze/services] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fSGYJ
[23:17:02] <Not-683e>	 [02miraheze/services] 07Wiki1776 0399f2b3d - add unidoswiki on services
[23:17:33] <MacFan4000>	 Wiki-1776: there you go :)
[23:19:26] <Wiki-1776>	 All right, thank you. You have already enabled VisualEditor? MacFan4000 
[23:19:45] <MacFan4000>	 I will do that now
[23:20:12] <Wiki-1776>	 Ok, thans
[23:21:02] <MacFan4000>	 Wiki-1776: Done
[23:21:15] <Wiki-1776>	 :)
[23:53:59] <Wiki-1776>	 I'm curious, if a file has 5-25 MB, how long does it take to import it into a wiki?
[23:55:50] <JohnLewis>	 genuinely a mystery
[23:56:23] <Wiki-1776>	 https://phabricator.miraheze.org/T3303
[23:56:25] <ZppixBot>	 Title: [ ⚓ T3303 Import for unidos.miraheze.org ] - phabricator.miraheze.org
[23:58:28] <JohnLewis>	 oh you're asking how long it will take us?
[23:58:41] <JohnLewis>	 genuinely an even bigger mystery
[23:59:19] <Wiki-1776>	 yes