[06:43:59] <RF1|pingonreply>	 paladox, Reception123: [ddac331a9a76ed66c8b91342] 2019-07-23 06:43:21: Fatal exception of type "MWException" on CW following a 504 can u check it?
[06:45:37] <RF1|pingonreply>	 wiki has gone through fine so could just be already exisits error following me refreshing after the 504
[06:46:25] <icinga-miraheze>	 PROBLEM - es4 Puppet on es4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[java]
[06:48:58] <RF1|pingonreply>	 SPF|Cloud: ^^^ see me above
[06:49:22] <RF1|pingonreply>	 and PuppyKun / @NDKilla
[06:54:25] <icinga-miraheze>	 RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:15:54] <Reception123>	 Will have no access until way later so it will have to be paladox 
[09:03:20] <paladox>	 RhinosF1 MWException from line 75 of /srv/mediawiki/w/extensions/CreateWiki/includes/WikiManager.php: Wiki 'mscwiki' already exists
[09:04:10] <RhinosF1>	 paladox: same error again, I'm guessing it created but returned the 504 HTTP to me and on refresh it's causing that. We probably should improve that error.
[09:07:37] <paladox>	 ok
[09:10:00] <Not-8a13>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjDIZ
[09:10:01] <Not-8a13>	 [02miraheze/mediawiki] 07paladox 03b1b370f - Revert "Update MirahezeMagic" This reverts commit 00586e8e6ce7aaf8d7c8491cfcda89a9b7982971.
[09:10:02] <paladox>	 I'm off again, RhinosF1 that ^^ may fix the issue for now.
[09:10:38] <RhinosF1>	 paladox: k
[09:12:27] <RhinosF1>	 paladox: https://phabricator.miraheze.org/T4567 for tracking
[09:12:28] <ZppixBot>	 [ ⚓ T4567 Improve CreateWiki already created wiki error ] - phabricator.miraheze.org
[10:02:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.94, 1.79, 1.10
[10:04:43] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.54, 1.33, 1.01
[10:05:12] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDLa
[10:05:13] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot 035c05150 - BOT: Updating services config for wikis
[10:24:44] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.70, 2.46, 1.53
[10:26:43] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 2.04, 2.34, 1.60
[10:34:43] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 3.56, 4.22, 2.78
[10:36:43] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.15, 2.98, 2.49
[10:53:49] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.23, 2.52, 1.64
[10:57:40] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.44, 1.49, 1.41
[11:03:27] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.15, 2.55, 1.89
[11:07:17] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.64, 1.83, 1.75
[11:09:12] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.00, 1.46, 1.62
[11:33:05] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 3.59, 2.81, 2.02
[11:35:05] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.51, 2.24, 1.90
[11:40:14] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDt9
[11:40:16] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot 03e909a20 - BOT: Updating services config for wikis
[11:43:09] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 5.81, 3.46, 2.39
[11:45:07] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.77, 2.67, 2.23
[12:03:18] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 6.33, 3.91, 2.59
[12:05:16] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 2.18, 3.10, 2.45
[12:37:54] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9131],Exec[ufw-allow-tcp-from-185.52.2.113-to-any-port-81],Exec[ufw-allow-tcp-from-81.4.121.113-to-any-port-81]
[13:02:30] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:02:46] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.27, 3.18, 1.84
[13:08:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.20, 1.97, 1.77
[13:12:43] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.97, 1.59, 1.67
[13:14:21] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:22:03] <RhinosF1>	 paladox: 504 on createwiki again
[13:22:52] <RhinosF1>	 paladox: [2b9c6f239b76bc65ea47786d] 2019-07-23 13:22:44: Fatal exception of type "MWException"
[13:23:06] <RhinosF1>	 now ^^ can you see if it's the usual?
[13:23:20] <paladox>	 ok
[13:24:05] <paladox>	 RhinosF1 you mention the same symptons as before. Meaning it's trying to recreate the wiki but will fail because the db already exists.
[13:24:50] <RhinosF1>	 paladox: yeah, we're getting a 504 on the first go so I'm refreshing the page and getting that, maybe we should look at why 504s are happening
[13:25:32] <paladox>	 JohnLewis can we expose why creating a wiki failed?
[13:25:43] <paladox>	 e.g RhinosF1 is getting an exception, but the logs says "MWException from line 75 of /srv/mediawiki/w/extensions/CreateWiki/includes/WikiManager.php: Wiki 'entecwikiwiki' already exists"
[13:26:03] <JohnLewis>	 paladox: everything except "wiki already exists" does
[13:26:11] <paladox>	 oh
[13:27:26] <paladox>	 I bet RhinosF1 is hitting the ES timeout due to the load.
[13:27:29] <RhinosF1>	 paladox: 
[13:27:42] <RhinosF1>	 paladox, JohnLewis: see https://phabricator.miraheze.org/T4568 for the 504s
[13:27:43] <ZppixBot>	 [ ⚓ T4568 repeated 504s on CreateWiki ] - phabricator.miraheze.org
[13:28:28] <JohnLewis>	 dropped CW project
[13:29:19] <RhinosF1>	 JohnLewis: k, put it on there because i've noticed the 504s happening on CW
[13:29:35] <JohnLewis>	 RhinosF1: strange as it sounds, it has nothing to do with CW in any possible way :)
[13:30:05] <RhinosF1>	 JohnLewis: okay, guessing based on paladox it's ES but I don't think that has a tag
[13:30:20] <JohnLewis>	 it does :)
[13:30:28] <RhinosF1>	 JohnLewis: what tag?
[13:30:35] <JohnLewis>	 ElasticSearch
[13:30:44] <RhinosF1>	 JohnLewis: shall I add it?
[13:30:48] <JohnLewis>	 sure
[13:32:14] <RhinosF1>	 JohnLewis: done and https://phabricator.miraheze.org/T4567 is for improving the CW error
[13:32:16] <ZppixBot>	 [ ⚓ T4567 Improve CreateWiki already created wiki error ] - phabricator.miraheze.org
[13:36:53] <Not-8a13>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjDYD
[13:36:54] <Not-8a13>	 [02miraheze/CreateWiki] 07JohnFLewis 039d6c507 - add try-catch
[13:37:25] <Not-8a13>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjDYy
[13:37:25] <JohnLewis>	 RhinosF1: done
[13:37:26] <Not-8a13>	 [02miraheze/mediawiki] 07JohnFLewis 0306bed76 - CW
[13:38:57] <RhinosF1>	 JohnLewis: looks good
[13:40:50] <RhinosF1>	 JohnLewis: works directly on CreateWiki fine but not on RequestWiki approve
[13:41:13] <Not-8a13>	 [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/fjDYb
[13:41:15] <Not-8a13>	 [02miraheze/MirahezeMagic] 07paladox 03a10817c - Create MirahezeMagicCreateWikiJob.php
[13:41:16] <Not-8a13>	 [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX
[13:41:18] <Not-8a13>	 [02MirahezeMagic] 07paladox opened pull request 03#85: Create MirahezeMagicCreateWikiJob.php - 13https://git.io/fjDYN
[13:41:25] <JohnLewis>	 RhinosF1: oh bah good catch
[13:44:32] <Not-8a13>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjDOU
[13:44:34] <Not-8a13>	 [02miraheze/CreateWiki] 07JohnFLewis 03f753a3f - add to RW too
[13:44:54] <Not-8a13>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjDOT
[13:44:56] <JohnLewis>	 RhinosF1: there you go
[13:44:56] <Not-8a13>	 [02miraheze/mediawiki] 07JohnFLewis 03e14dba2 - CW
[13:47:44] <RhinosF1>	 JohnLewis: no change yet testing on the wiki I just created will try again in 10-15 mins to give it time to deploy
[13:48:06] <JohnLewis>	 yeah it won't have deployed yet
[13:51:24] <JohnLewis>	 RhinosF1: https://phabricator.miraheze.org/T4567#86771 'tis minor but if I'm around, don't reopen tasks unless I say so - just creates more noise :)
[13:51:25] <ZppixBot>	 [ ⚓ T4567 Improve CreateWiki already created wiki error ] - phabricator.miraheze.org
[13:51:37] <RhinosF1>	 JohnLewis: okay
[14:02:19] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.37, 2.99, 2.00
[14:16:45] <RhinosF1>	 JohnLewis: looks good!!
[14:16:52] <JohnLewis>	 good good
[14:25:54] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:37:55] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9131]
[14:38:51] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.47, 1.40, 1.93
[14:40:00] <Not-8a13>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjD3Y
[14:40:01] <Not-8a13>	 [02miraheze/dns] 07paladox 0348d1f20 - Depool cp3
[14:40:14] <paladox>	 !log depool cp3
[14:40:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:44:43] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.81, 1.14, 1.65
[14:45:21] <paladox>	 !log increasing swap on cp3 to 2gb
[14:45:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:46:03] <paladox>	 !log reboot cp3
[14:46:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:47:59] <Not-8a13>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjD3R
[14:48:00] <Not-8a13>	 [02miraheze/dns] 07paladox 03c47d438 - Revert "Depool cp3" This reverts commit 48d1f20bb383cf255a41d1878974753058965fcd.
[14:49:25] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2645 MB (10% inode=94%);
[14:55:53] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:02:46] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 5.45, 3.04, 1.74
[15:02:51] <icinga-miraheze>	 PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 3.80, 2.68, 1.80
[15:03:06] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.96, 2.69, 1.95
[15:04:51] <icinga-miraheze>	 RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 3.15, 2.87, 1.98
[15:12:51] <icinga-miraheze>	 PROBLEM - misc4 Current Load on misc4 is WARNING: WARNING - load average: 2.29, 3.54, 2.63
[15:14:51] <icinga-miraheze>	 RECOVERY - misc4 Current Load on misc4 is OK: OK - load average: 1.06, 2.73, 2.44
[15:20:43] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 0.72, 2.93, 3.46
[15:22:43] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.41, 2.41, 3.20
[15:26:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.39, 1.36, 1.99
[15:28:16] <paladox>	 !log apt upgrade - test1
[15:28:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:30:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.27, 1.73, 1.93
[15:32:44] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.98, 1.67, 1.88
[15:32:52] <icinga-miraheze>	 PROBLEM - es4 Puppet on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[15:32:56] <icinga-miraheze>	 PROBLEM - es3 SSH on es3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:33:04] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 8.64, 6.44, 4.57
[15:33:42] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 8.30, 7.00, 3.54
[15:33:56] <icinga-miraheze>	 PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[15:34:20] <icinga-miraheze>	 PROBLEM - es1 Puppet on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[15:34:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.22, 2.41, 2.13
[15:34:57] <icinga-miraheze>	 RECOVERY - es3 SSH on es3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0)
[15:35:06] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 6.76, 4.57, 2.91
[15:35:53] <icinga-miraheze>	 RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures
[15:36:19] <icinga-miraheze>	 RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures
[15:36:56] <icinga-miraheze>	 RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures
[15:38:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.42, 1.50, 1.83
[15:42:43] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.12, 1.31, 1.67
[15:42:43] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 1.41, 3.09, 3.91
[15:43:06] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 2.02, 3.74, 3.44
[15:45:05] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 0.72, 2.74, 3.11
[15:45:25] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 1.67, 3.23, 3.59
[15:46:43] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.66, 1.90, 3.25
[15:47:22] <icinga-miraheze>	 RECOVERY - es4 Current Load on es4 is OK: OK - load average: 1.23, 2.64, 3.34
[15:47:52] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.12, 0.51, 1.84
[15:48:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.59, 1.56, 1.72
[15:49:52] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.23, 0.43, 1.65
[15:50:43] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.31, 1.47, 1.66
[15:55:14] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 3.99, 3.30, 3.24
[15:55:40] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.84, 3.58, 3.39
[15:57:36] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.87, 3.67, 3.45
[15:59:31] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.42, 3.95, 3.58
[16:01:06] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 7.41, 4.79, 3.80
[16:04:17] <Not-8a13>	 [02mw-config] 07The-Voidwalker opened pull request 03#2706: wgApprovedRevs -> egApprovedRevs - 13https://git.io/fjDsN
[16:04:59] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 1.49, 3.54, 3.59
[16:05:18] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 0.80, 3.41, 3.66
[16:06:55] <Not-8a13>	 [02mw-config] 07paladox closed pull request 03#2706: wgApprovedRevs -> egApprovedRevs - 13https://git.io/fjDsN
[16:06:56] <icinga-miraheze>	 RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.79, 2.71, 3.28
[16:06:56] <Not-8a13>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjDGf
[16:06:58] <Not-8a13>	 [02miraheze/mw-config] 07The-Voidwalker 037dbb2d8 - wgApprovedRevs -> egApprovedRevs (#2706) extension uses eg prefix instead of wg prefix
[16:07:13] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.44, 2.43, 3.27
[16:09:05] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.56, 3.16, 2.47
[16:11:05] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 3.00, 3.13, 2.55
[16:32:44] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 5.66, 3.32, 2.51
[16:35:05] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 5.01, 3.84, 2.64
[16:37:05] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.64, 2.95, 2.46
[16:42:43] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 1.34, 3.51, 3.17
[16:44:43] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.62, 2.53, 2.85
[17:02:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.94, 1.84, 1.05
[17:03:10] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.86, 3.36, 3.04
[17:04:43] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.72, 1.50, 1.02
[17:05:05] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 3.24, 3.30, 3.05
[17:17:36] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 6.86, 9.04, 6.89
[17:19:35] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 3.16, 6.93, 6.38
[17:21:33] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 2.42, 5.48, 5.91
[17:33:57] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.55, 3.27, 2.79
[17:34:22] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 5.57, 3.84, 2.82
[17:35:52] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 4.00, 3.60, 2.98
[17:38:21] <icinga-miraheze>	 RECOVERY - es4 Current Load on es4 is OK: OK - load average: 1.58, 3.28, 2.88
[17:39:49] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 5.03, 4.23, 3.36
[17:43:08] <icinga-miraheze>	 PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:45:06] <icinga-miraheze>	 RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures
[17:52:24] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 5.51, 4.13, 3.30
[17:52:51] <icinga-miraheze>	 PROBLEM - es4 Puppet on es4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:54:49] <icinga-miraheze>	 RECOVERY - es4 Puppet on es4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[18:14:21] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is WARNING: WARNING - load average: 0.20, 2.12, 3.54
[18:16:21] <icinga-miraheze>	 RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.07, 1.44, 3.11
[18:18:43] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 0.65, 1.99, 3.70
[18:22:43] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.87, 1.84, 3.23
[18:35:12] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDnH
[18:35:13] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot 037bbf936 - BOT: Updating services config for wikis
[18:56:21] <icinga-miraheze>	 RECOVERY - Host db5 is UP: PING OK - Packet loss = 73%, RTA = 78.60 ms
[18:56:58] <icinga-miraheze>	 PROBLEM - db5 MySQL on db5 is CRITICAL: Can't connect to MySQL server on '168.235.110.50' (111 "Connection refused")
[19:03:10] <icinga-miraheze>	 RECOVERY - db5 SSH on db5 is OK: SSH OK - OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 (protocol 2.0)
[19:14:29] <paladox>	 bah
[19:14:35] * paladox goes into progress to remove db5
[19:17:36] <paladox>	 *postgres
[20:22:47] <RhinosF1>	 paladox: https://meta.miraheze.org/w/index.php?title=Community_noticeboard&curid=6947&diff=78465&oldid=78461 -- is that u breaking things?
[20:22:49] <ZppixBot>	 [ Difference between revisions of "Community noticeboard" - Miraheze Meta ] - meta.miraheze.org
[20:26:22] <paladox>	 hmm
[20:26:37] <paladox>	 i doin't think so.
[20:26:52] <RhinosF1>	 paladox: then what's happened???
[20:27:27] <paladox>	 I'm not sure.
[20:27:37] <RhinosF1>	 i'll file a task
[20:31:03] <RhinosF1>	 paladox: ^ done in https://phabricator.miraheze.org/T4569
[20:31:05] <ZppixBot>	 [ ⚓ T4569 Special:Analytics is missing most fields ] - phabricator.miraheze.org
[20:31:08] <paladox>	 ok
[20:33:27] <RhinosF1>	 paladox: matomo's repo hasn't changed in 21 days. is everything on db4?
[20:33:34] <paladox>	 yup.
[20:34:21] <RhinosF1>	 paladox: strange, I can't see what could have happened as nothing should have changed then
[20:36:33] <paladox>	 !log sudo -u www-data /usr/local/bin/fileLockScript.sh /tmp/matomo_file_lock "/usr/bin/nice -19 /usr/bin/php /srv/matomo/console core:archive --url=https://matomo.miraheze.org/" > /srv/matomo-archive.log' - misc2
[20:36:34] <ZppixBot>	 [ 404 Not Found ] - matomo.miraheze.org
[20:36:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:38:07] <paladox>	 RhinosF1 i think ^^ will fix it.
[20:38:19] <RhinosF1>	 paladox: let's see, how long to deploy?
[20:38:34] <paladox>	 deploy? It's a script so no deploying needed :)
[20:38:38] <paladox>	 secondly it may take hours
[20:39:16] <RhinosF1>	 paladox: well show then, u know what i mean. and no change yet
[20:39:44] <paladox>	 ok
[21:06:23] <Not-8a13>	 [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_33 [+0/-0/±2] 13https://git.io/fjDWX
[21:06:24] <Not-8a13>	 [02miraheze/mediawiki] 07paladox 038975de0 - Update ArticleFeedbackv5
[21:06:26] <Not-8a13>	 [02miraheze/mediawiki] 07paladox 031858695 - Update ArticleFeedbackv5
[21:28:00] <paladox>	 !log restart prometheus - phd - php-fpm on misc (OOM)
[21:28:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:28:11] <paladox>	 !log that was for misc4
[21:28:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:28:33] <paladox>	 !log upgrade puppet-agent - misc4
[21:28:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:31:19] <Not-8a13>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDlv
[21:31:21] <Not-8a13>	 [02miraheze/puppet] 07paladox 036ce83c0 - Reduce salt master workers to 4
[21:32:02] <MH-Discord>	 <RhinosF1> paladox: can u give https://phabricator.miraheze.org/T4562 a look?
[21:32:03] <ZppixBot>	 [ ⚓ T4562 Custom Domain and SSL Certificate for Wiesepedia ] - phabricator.miraheze.org
[21:32:17] <paladox>	 yup
[21:33:23] <Not-8a13>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjDlU
[21:33:25] <Not-8a13>	 [02miraheze/dns] 07paladox 03c6a79d7 - Add wiesepedia.ga to dns
[21:33:36] <RhinosF1>	 paladox: they're @GOTILON#6175  on discord if you need anything
[21:34:12] <paladox>	 ok
[21:35:43] <paladox>	 !log root@misc4:/srv# rm /etc/aliases - puppet failing with " Could not prefetch mailalias provider 'aliases': Could not parse line "postmaster: root" (file: /etc/aliases, line: 3)"
[21:35:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:39:28] <Not-8a13>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDlY
[21:39:30] <Not-8a13>	 [02miraheze/puppet] 07paladox 036ff17b7 - Revert "Reduce salt master workers to 4" This reverts commit 6ce83c0386a1827e91e68353806d9d0abba8678a.
[21:42:17] <paladox>	 !log upgrade puppet-agent, puppetdb and puppetserver on puppet1
[21:42:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:45:39] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown)
[21:46:32] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 117 failures. Last run 2 minutes ago with 117 failures. Failed resources (up to 3 shown)
[21:46:42] <icinga-miraheze>	 PROBLEM - es3 Puppet on es3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:46:45] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown)
[21:47:25] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown)
[21:47:40] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 189 failures. Last run 3 minutes ago with 189 failures. Failed resources (up to 3 shown): File[wiki.campaign-labour.org_private],File[wiki.bonessdiving.club],File[wiki.bonessdiving.club_private],File[wiki.rmbrk.sk]
[21:48:18] <Not-8a13>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:48:20] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDlZ
[21:48:21] <Not-8a13>	 [02miraheze/ssl] 07paladox 030147408 - Add wiesepedia.ga ssl certificate
[21:48:23] <Not-8a13>	 [02ssl] 07paladox opened pull request 03#205: Add wiesepedia.ga ssl certificate - 13https://git.io/fjDln
[21:48:52] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjDlc
[21:48:53] <Not-8a13>	 [02miraheze/ssl] 07paladox 03b138b8d - Create wiesepedia.ga.crt
[21:49:34] <Not-8a13>	 [02ssl] 07paladox closed pull request 03#205: Add wiesepedia.ga ssl certificate - 13https://git.io/fjDln
[21:49:36] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDlC
[21:49:37] <Not-8a13>	 [02miraheze/ssl] 07paladox 0339ebe3f - Add wiesepedia.ga ssl certificate (#205)
[21:49:43] <Not-8a13>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:49:44] <Not-8a13>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[21:50:37] <icinga-miraheze>	 PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 3.42, 2.39, 1.57
[21:51:42] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.48, 2.73, 1.71
[21:52:36] <icinga-miraheze>	 RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.55, 1.98, 1.52
[21:52:52] <Not-8a13>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjDl8
[21:52:54] <Not-8a13>	 [02miraheze/dns] 07paladox 030b29c00 - Add tradwiki.org to dns - T4472
[21:55:12] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDlu
[21:55:13] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot 03c5f7c3e - BOT: Updating services config for wikis
[21:55:31] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:55:39] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:55:42] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 2.94, 3.30, 2.21
[21:55:50] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[21:55:52] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:56:17] <Not-8a13>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:56:18] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDl2
[21:56:20] <Not-8a13>	 [02miraheze/ssl] 07paladox 036e54f32 - Add tradwiki.org ssl certificate Redirects tradwiki.org too www.tradwiki.org. Also adds www.tradwiki.org.
[21:56:21] <Not-8a13>	 [02ssl] 07paladox opened pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[21:56:28] <icinga-miraheze>	 PROBLEM - es2 Puppet on es2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:56:45] <icinga-miraheze>	 RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:57:15] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDlV
[21:57:17] <Not-8a13>	 [02miraheze/ssl] 07paladox 038277e32 - Update redirects.yaml
[21:57:18] <Not-8a13>	 [02ssl] 07paladox synchronize pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[21:57:23] <icinga-miraheze>	 PROBLEM - es1 Puppet on es1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:57:39] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDlw
[21:57:39] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:57:40] <Not-8a13>	 [02miraheze/ssl] 07paladox 03cbd1edb - Update certs.yaml
[21:57:42] <Not-8a13>	 [02ssl] 07paladox synchronize pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[21:58:45] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjDlo
[21:58:47] <Not-8a13>	 [02miraheze/ssl] 07paladox 0320f1696 - Create tradwiki.org.crt
[21:58:48] <Not-8a13>	 [02ssl] 07paladox synchronize pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[21:59:40] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.56, 3.57, 2.57
[22:01:20] <Not-8a13>	 [02ssl] 07paladox synchronize pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[22:01:21] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDli
[22:01:23] <Not-8a13>	 [02miraheze/ssl] 07paladox 032beeece - Update certs.yaml
[22:01:38] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.98, 3.91, 2.80
[22:01:39] <Not-8a13>	 [02ssl] 07paladox synchronize pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[22:01:41] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDlP
[22:01:42] <Not-8a13>	 [02miraheze/ssl] 07paladox 03b43dfa4 - Update redirects.yaml
[22:01:57] <Not-8a13>	 [02ssl] 07paladox synchronize pull request 03#206: Add tradwiki.org ssl certificate - 13https://git.io/fjDla
[22:01:59] <Not-8a13>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjDlX
[22:02:00] <Not-8a13>	 [02miraheze/ssl] 07paladox 032fb9fa0 - Update tradwiki.org.crt
[22:03:13] <icinga-miraheze>	 RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[22:04:07] <icinga-miraheze>	 RECOVERY - es2 Puppet on es2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:05:13] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjDl1
[22:05:15] <Not-8a13>	 [02miraheze/services] 07MirahezeSSLBot 03ebba533 - BOT: Updating services config for wikis
[22:05:32] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 2.35, 3.60, 2.97
[22:07:29] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.19, 2.70, 2.71
[22:27:16] <Not-8a13>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjD8I
[22:27:18] <Not-8a13>	 [02miraheze/mediawiki] 07paladox 0366ebd80 - Update Maps
[22:42:23] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 5.54, 2.97, 1.75
[22:44:23] <icinga-miraheze>	 RECOVERY - es4 Current Load on es4 is OK: OK - load average: 2.60, 2.80, 1.85
[22:48:27] <icinga-miraheze>	 PROBLEM - es4 Current Load on es4 is CRITICAL: CRITICAL - load average: 7.12, 5.02, 2.95
[22:48:29] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 5.64, 3.83, 2.70
[22:56:23] <icinga-miraheze>	 RECOVERY - es4 Current Load on es4 is OK: OK - load average: 0.52, 2.93, 2.97
[23:08:26] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 1.57, 3.10, 3.69
[23:12:26] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.87, 2.28, 3.20
[23:33:04] <icinga-miraheze>	 PROBLEM - test1 Disk Space on test1 is WARNING: DISK WARNING - free space: / 4502 MB (10% inode=98%);
[23:35:05] <icinga-miraheze>	 PROBLEM - es2 Current Load on es2 is WARNING: WARNING - load average: 3.41, 3.03, 2.33
[23:36:27] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 4.69, 3.64, 2.72
[23:37:04] <icinga-miraheze>	 RECOVERY - es2 Current Load on es2 is OK: OK - load average: 0.78, 2.18, 2.10
[23:40:26] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.42, 3.56, 2.90
[23:42:26] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 8.46, 4.62, 3.33
[23:54:26] <icinga-miraheze>	 PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 1.28, 3.67, 3.78
[23:56:26] <icinga-miraheze>	 RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.44, 2.58, 3.36