[00:11:20] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-7/±0] 13https://git.io/fhAaG
[00:11:22] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0345105a5 - Remove unused parsoid class This is now in the services class.
[00:17:11] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAac
[00:17:12] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0340b8c07 - Icinga2: Enable mysql ssl in idomysql
[00:20:26] <icinga-miraheze>	 CUSTOM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 38452 MB (10% inode=96%); paladox test
[00:20:58] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAa4
[00:21:00] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 38452 MB (10% inode=96%);
[00:21:00] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03c2bc7c4 - icinga2: Fix ssl vars for idomysql
[00:42:21] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fhAaM
[00:42:22] <Not-8d6a>	 [02miraheze/puppet] 07paladox 034d7070a - Revert "icinga2: Fix ssl vars for idomysql" This reverts commit c2bc7c42b7e8f27278cc813904f0d51ce38f685e.
[00:42:24] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03e55d3c5 - Revert "Icinga2: Enable mysql ssl in idomysql" This reverts commit 40b8c07998fe0f1c56599313d3dc7db194258d9a.
[01:07:09] <SPF|Cloud>	 !log ensure presence of iperf, vpncloud and net-tools packages on cp5 and test1
[01:07:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:14:55] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03icinga2-upgrade [+111/-1/±29] 13https://git.io/fhAVG
[01:14:57] <Not-8d6a>	 [02miraheze/puppet] 07paladox 039f3cecd - icinga2: Update module to version 2.0.1
[01:14:58] <Not-8d6a>	 [02puppet] 07paladox created branch 03icinga2-upgrade - 13https://git.io/vbiAS
[01:20:29] <SPF|Cloud>	 !log previous entry also for db4
[01:20:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:44:54] <icinga-miraheze>	 PROBLEM - misc1 Check correctness of the icinga configuration on misc1 is CRITICAL: Icinga configuration contains errors
[01:45:04] <paladox>	 know ^^
[01:45:08] <paladox>	 *known
[01:49:10] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03icinga2-upgrade [+110/-1/±31] 13https://git.io/fhAwf
[01:49:11] <Not-8d6a>	 [02miraheze/puppet] 07paladox 036b4dbce - icinga2: Update module to version 2.0.1
[01:52:55] <icinga-miraheze>	 RECOVERY - misc1 Check correctness of the icinga configuration on misc1 is OK: Icinga configuration is correct
[01:57:36] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03icinga2-upgrade [+110/-1/±31] 13https://git.io/fhAwm
[01:57:38] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0362c7fe6 - icinga2: Update module to version 2.0.1
[02:00:44] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[icinga2 pki get trusted-cert]
[02:05:02] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[02:05:10] <SPF|Cloud>	 !log graceful vpncloud shutdown
[02:05:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:05:26] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03icinga2-upgrade [+110/-1/±31] 13https://git.io/fhAwn
[02:05:28] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0331cfbec - icinga2: Update module to version 2.0.1
[02:07:01] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+110/-1/±31] 13https://git.io/fhAwW
[02:12:47] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[idomysql-import-schema]
[02:47:40] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:39:17] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhArV
[03:39:19] <Not-8d6a>	 [02miraheze/puppet] 07paladox 037214b6c - icinga2: Enable ssl and disable import_schema
[04:38:32] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:40:27] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000.
[06:13:20] <Reception123>	 !log reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki absurdopediawiki /home/reception/absurdopedia_reverted_pages_full.xml
[06:13:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[09:32:09] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 795
[09:40:50] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:42:48] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[09:54:09] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[10:22:33] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 720
[11:14:09] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[11:22:09] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 646
[11:28:07] <ZppixBot>	 Hello Aldnonymous! If you have any questions feel free to ask and someone should answer soon.
[11:38:09] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[13:10:51] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 419
[13:18:38] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[13:30:38] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:32:33] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000.
[13:55:10] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAMJ
[13:55:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 038d75491 - BOT: Updating services config for wikis
[14:11:30] <Not-8d6a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAMn
[14:11:31] <Not-8d6a>	 [02miraheze/mw-config] 07Reception123 035a467a6 - compress absurdopediawiki revisions (large import) per Paladox
[14:14:04] <Reception123>	 !log reception@mw3:/srv/mediawiki/w/maintenance/storage$ sudo -u www-data php compressOld.php --wiki absurdopediawiki -t = 'gzip'
[14:14:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:18:48] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAM4
[14:18:50] <Not-8d6a>	 [02miraheze/puppet] 07paladox 032074ee0 - Reduce php-fpm childs
[14:19:04] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAMB
[14:19:06] <Not-8d6a>	 [02miraheze/puppet] 07paladox 035481c2a - Increase max_execution_time to 100
[14:59:09] <paladox>	 !log upgraded grafana to version 6 and icinga2 to 2.10.3 on misc1
[14:59:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:24:06] <paladox>	 !log upgraded salt on bacula1
[15:24:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:36:01] <paladox>	 !log upgrade puppet1 to debian 9.8
[15:36:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:44:03] <paladox>	 !log installing security updates on db4, mw*
[15:44:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:47:37] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:49:36] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 10 May 2019 08:37:39 PM GMT +0000.
[16:46:10] <paladox>	 !log stop miraheze ssl renewal service on mw1 && disable puppet
[16:46:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:49:48] <icinga-miraheze>	 PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: connect to address 185.52.1.75 and port 5000: Connection refused
[16:50:05] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: Upgrading our ssl infrastructure, last run 7 minutes ago with 0 failures
[16:59:12] <paladox>	 !log mkdir /mnt/mediawiki-static/private/miraheze && /mnt/mediawiki-static/private/miraheze/letsencrypt
[16:59:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:14:48] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fhA9l
[18:14:50] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 03c3077c8 - add 3 new variables to MatomoAnalytics Operating system Version, resolution type, unique visits
[18:15:48] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhA9R
[18:15:50] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 0386a26c8 - minor fix
[18:17:23] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhA9u
[18:17:24] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 03889dd56 - Update MatomoAnalytics
[18:19:36] <Reception123>	 paladox: heh the problem is now we're going to have a little inbalance since puppet will not run on mw1 (so when the request is served by mw1 users won't see the new Matomo change)
[18:20:08] <paladox>	 Reception123 you can run cd /srv/med*/w && sudo -u www-data git pull && sudo -u www-data git submodule update
[18:20:45] <Reception123>	 ok
[18:21:02] <Reception123>	 don't like not using Puppet but I guess it's better than different content on the servers
[18:21:25] <Reception123>	 !log reception@mw1:~$ cd /srv/med*/w && sudo -u www-data git pull && sudo -u www-data git submodule update (Puppet disabled on mw1, inbalance created by change)
[18:21:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:28:37] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhA9o
[18:28:39] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07paladox 037057a0d - fix $statearray -> $statarray
[18:30:13] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhA9i
[18:30:15] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 03926c1c5 - Update MatomoAnalytics
[18:32:12] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhA9D
[18:32:13] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 0356343aa - fix another typo
[18:32:54] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhA9y
[18:32:55] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 03be47714 - Update MatomoAnalytics
[18:35:49] <icinga-miraheze>	 PROBLEM - www.netazar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.netazar.org' expires in 15 day(s) (Mon 18 Mar 2019 06:33:35 PM GMT +0000).
[18:36:08] <icinga-miraheze>	 PROBLEM - netazar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.netazar.org' expires in 15 day(s) (Mon 18 Mar 2019 06:33:35 PM GMT +0000).
[18:36:18] <paladox>	 oh now they decide they need renewing
[18:37:16] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhA9d
[18:37:17] <Not-8d6a>	 [02miraheze/MatomoAnalytics] 07Reception123 03826f617 - one last typo I agree that this is kind of ridiculous...
[18:38:26] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhA9A
[18:38:27] <Not-8d6a>	 [02miraheze/mediawiki] 07Reception123 0324debba - Update MatomoAnalytics
[18:50:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhAHU
[18:50:12] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03c2cee49 - BOT: Updating services config for wikis
[18:53:15] <Hispano76>	 hi
[19:49:31] <paladox>	 hi
[21:03:48] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 45907 MB (12% inode=96%);
[21:05:10] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhA7a
[21:05:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 037b483ca - BOT: Updating services config for wikis
[22:24:22] <Zppix>	 Voidwalker: Do i have to do the template for denied wiki requests still or no?
[22:24:48] <Voidwalker>	 you have to notify them somehow if you decline their request
[22:25:16] <Zppix>	 Voidwalker: ok wasnt sure I didnt want to break or do anything thats not needed
[22:27:01] <Zppix>	 Ugh i hate using kiwiirc on my phone it disconnects if i tab out of it for extended periods
[22:27:18] <Voidwalker>	 tbh though, I didn't think you could submit a wiki request using an empty subdomain
[22:27:46] <Zppix>	 Voidwalker: he edited it after he created it
[22:28:01] <Zppix>	 voidwalker: maybe its an error on the entry verificaiton
[22:28:10] <Zppix>	 verification*
[22:28:14] <Voidwalker>	 hmm, guess we need to adjust the edit request form thinggy
[22:28:41] <Zppix>	 Voidwalker: Are you sure you shouldnt adjust the thingabob?
[22:30:12] <Voidwalker>	 hmm, it's either the doohicky or the jim-jam
[22:30:41] <Zppix>	 Voidwalker: No i think its that thingamajig
[22:34:04] <Not-8d6a>	 [02CreateWiki] 07The-Voidwalker opened pull request 03#111: require form fields to be filled - 13https://git.io/fhA51
[22:35:01] <Zppix>	 Voidwalker: I also think i fixed ZppixBot’s disconnect issue (well i had help)
[22:35:19] <Voidwalker>	 hey, that's good
[22:35:39] <Not-8d6a>	 [02CreateWiki] 07paladox closed pull request 03#111: require form fields to be filled - 13https://git.io/fhA51
[22:35:40] <Not-8d6a>	 [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhA5S
[22:35:42] <Not-8d6a>	 [02miraheze/CreateWiki] 07The-Voidwalker 037b6abe3 - require form fields to be filled (#111)
[22:35:53] <Zppix>	 Voidwalker: it was some upstream module that was added in called instgram and once we disabled it it appeared to fix it
[22:35:55] <Zppix>	 .uptime
[22:35:55] <ZppixBot>	 I've been sitting here for 4 days, 2:44:27 and I keep going!
[22:54:11] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhAdI
[22:54:12] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03d6f1cef - Update CreateWiki
[22:54:18] <paladox>	 Voidwalker ^^
[23:22:17] <icinga-miraheze>	 PROBLEM - spiral.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:24:22] <icinga-miraheze>	 RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Wed 08 May 2019 09:46:50 PM GMT +0000.
[23:48:09] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 441
[23:50:09] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs