[00:30:17] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.009 second response time
[00:32:14] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.019 second response time
[00:47:52] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.007 second response time
[00:50:30] <Not-7290>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[00:50:31] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjPSH
[00:50:33] <Not-7290>	 [02miraheze/puppet] 07paladox 03c7f42b3 - elasticsearch: Raise heap to 2g
[00:50:34] <Not-7290>	 [02puppet] 07paladox opened pull request 03#1027: elasticsearch: Raise heap to 2g - 13https://git.io/fjPSQ
[00:51:45] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.012 second response time
[00:52:05] <Not-7290>	 [02puppet] 07paladox closed pull request 03#1027: elasticsearch: Raise heap to 2g - 13https://git.io/fjPSQ
[00:52:06] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPS7
[00:52:08] <Not-7290>	 [02miraheze/puppet] 07paladox 03c2734be - elasticsearch: Raise heap to 2g (#1027)
[00:52:16] <Not-7290>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[00:52:18] <Not-7290>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[01:53:40] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[01:55:35] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 hours ago with 0 failures
[01:56:58] <icinga-miraheze>	 RECOVERY - lizardfs3 Puppet on lizardfs3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:57:00] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:57:06] <icinga-miraheze>	 RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:57:07] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:57:08] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:57:13] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:57:30] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:57:57] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:58:03] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:58:07] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:58:18] <icinga-miraheze>	 RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:58:19] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:58:27] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:58:37] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[07:53:08] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:15:24] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.038 second response time
[09:33:01] <Reception123>	 !log purge binary logs before '2019-07-12 09:00:00';
[09:33:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[09:35:19] <Not-7290>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjP5Q
[09:35:20] <Not-7290>	 [02miraheze/dns] 07Reception123 0358ef526 - add mrlove.wiki zone
[09:38:57] <Not-7290>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjP5x
[09:38:59] <Not-7290>	 [02miraheze/ssl] 07Reception123 03fe4a40d - add mrlove.wiki cert
[09:45:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPdk
[09:45:14] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 0309a8ffa - BOT: Updating services config for wikis
[09:52:28] <Reception123>	 !log reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importImages.php --wiki wackypediawiki --search-recursively /home/reception/wackypediaimages
[09:52:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[09:53:26] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wackypediawiki --report=1 /home/reception/illogicopedia_pages_full.xml
[09:53:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[10:14:57] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:29:37] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 5.018 second response time
[11:56:07] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.86, 6.70, 5.48
[11:58:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.77, 6.88, 5.68
[12:00:06] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.53, 6.70, 5.76
[12:23:23] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPNJ
[12:23:25] <Not-7290>	 [02miraheze/mw-config] 07paladox 03acd2b36 - Remove Sitenotice Change-Id: I5046c930b6eb3fcb388b42391e3a431bb6a1aa60
[12:29:00] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPNO
[12:29:02] <Not-7290>	 [02miraheze/puppet] 07paladox 03a9538f9 - elasticsearch: Raise heap to 2.5g
[12:35:17] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPNn
[12:35:19] <Not-7290>	 [02miraheze/puppet] 07paladox 0367d9f0b - elasticsearch: fix heap
[12:35:39] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.007 second response time
[12:39:39] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.010 second response time
[13:04:31] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjPA8
[13:04:32] <Not-7290>	 [02miraheze/mediawiki] 07paladox 0312222ff - Update Tweeki
[13:06:00] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPAR
[13:06:02] <Not-7290>	 [02miraheze/mw-config] 07paladox 036a6a079 - Add isset check for $wmgSearchType and $wmgDisableSearchUpdate
[13:38:22] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPxl
[13:38:24] <Not-7290>	 [02miraheze/mw-config] 07paladox 035f6c475 - Revert "Add isset check for $wmgSearchType and $wmgDisableSearchUpdate" This reverts commit 6a6a079e4577e028cf78da56aed7fc62c6b99707.
[14:38:30] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPpA
[14:38:32] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis 03e87a0d5 - fix common case where requirements can be voided
[14:45:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjPh3
[14:45:13] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03e4fd5bf - BOT: Updating services config for wikis
[15:22:21] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.03, 6.57, 5.87
[15:26:20] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.27, 6.67, 6.09
[15:30:19] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 7.02, 6.37
[15:32:18] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.27, 6.76, 6.35
[15:36:28] <RhinosF1>	 is it only me that think znotch190711 is proposing way too many rfcs and getting disruptive now?
[15:39:21] <RhinosF1>	 meta is down 502 bad gateway
[15:39:33] <RhinosF1>	 JohnLewis, paladox, Reception123: ^^
[15:39:55] <paladox>	 meta's up to me
[15:40:10] <RhinosF1>	 paladox: back weird.
[15:40:18] <paladox>	 yup
[15:40:27] <Reception123>	 RhinosF1: I agree about the disruption 
[15:40:42] <Reception123>	 In these cases I believe the correct procedure is a ban request at AN
[15:40:56] <Reception123>	 Since disruption is more of a community thing to decide (at least that's what we did in the past)
[15:41:15] <Reception123>	 Then the community decides and votes on what should be done 
[15:41:24] <RhinosF1>	 Reception123: I'll do it. He's also lying at times and I'm not that confident about his hypothetical meta, personally it's a bit too much
[15:42:20] <Reception123>	 Haven't seen the latest, will look
[15:42:45] <RhinosF1>	 Reception123: latest is a massive page on cvt
[15:43:15] <Reception123>	 RhinosF1: yeah just saw. A similar RfC has already been proposed and rejected 
[15:43:49] <Reception123>	 If you propose a community ban I will comment there 
[15:44:02] <Reception123>	 But even if it is in good faith there is a lot of disruption 
[15:44:13] <RhinosF1>	 Reception123: typing it up now
[15:44:23] <Reception123>	 And a waste of peoples time 
[15:44:26] <Reception123>	 RhinosF1: Ok 
[15:51:14] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.35, 6.75, 6.42
[15:55:14] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.39, 6.44, 6.40
[15:58:32] <RhinosF1>	 Reception123: https://meta.miraheze.org/wiki/Meta:Administrators%27_noticeboard#Znotch190711
[15:58:33] <ZppixBot>	 [ Meta:Administrators' noticeboard - Miraheze Meta ] - meta.miraheze.org
[16:01:09] <Reception123>	 Ok. I will leave it to other users for the time being and will comment if necessary 
[16:03:45] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.42, 6.20, 4.49
[16:05:45] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.06, 6.25, 4.74
[16:08:56] <RhinosF1>	 Reception123: thx
[16:09:10] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.65, 6.07, 5.88
[16:11:09] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.72, 7.19, 6.30
[16:11:54] <Reception123>	 RhinosF1: https://meta.miraheze.org/w/index.php?title=Meta:Administrators%27_noticeboard&diff=77771&oldid=77769
[16:11:55] <ZppixBot>	 [ Difference between revisions of "Meta:Administrators' noticeboard" - Miraheze Meta ] - meta.miraheze.org
[16:17:08] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.19, 7.91, 6.99
[16:17:12] <RhinosF1>	 Reception123: replied
[16:31:05] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.43, 5.82, 6.51
[16:39:02] <Reception123>	 RhinosF1: replied back :)
[16:39:17] <Reception123>	 It's more of a technicality but still valid 
[16:44:03] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.23, 6.39, 6.24
[16:46:03] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.10, 6.14, 6.16
[16:51:39] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.012 second response time
[16:54:49] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXeD
[16:54:51] <Not-7290>	 [02miraheze/puppet] 07paladox 0355179f9 - elasticsearch: Increase heap to 3G
[16:55:39] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.010 second response time
[16:56:02] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.46, 7.12, 6.49
[16:58:02] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.71, 7.31, 6.63
[17:04:02] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.45, 6.65, 6.59
[17:08:01] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.20, 8.44, 7.31
[17:24:03] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.19, 7.52, 7.75
[17:30:07] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.64, 7.63, 7.65
[17:32:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.96, 7.16, 7.46
[17:40:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.34, 7.49, 7.37
[17:41:43] <MH-Discord>	 <RhinosF1> testing 123.. testing
[17:41:50] <RhinosF1>	 test this way
[17:42:26] <MH-Discord>	 <RhinosF1> nope - just my discord broke not the bot
[17:44:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.78, 7.66, 7.46
[17:49:43] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±2] 13https://git.io/fjXvz
[17:49:45] <Not-7290>	 [02miraheze/mediawiki] 07paladox 03d0f948f - Update Comments
[17:50:38] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±2] 13https://git.io/fjXvr
[17:50:39] <Not-7290>	 [02miraheze/mediawiki] 07paladox 0397a9838 - Update VoteNY
[17:50:45] <paladox>	 JohnLewis ^
[17:51:20] <RhinosF1>	 paladox: is search still down?
[17:51:32] <paladox>	 RhinosF1 i think so, es crashed..
[17:51:43] <paladox>	 it's recovering 
[17:52:03] <RhinosF1>	 paladox: okay, just wondered as its just failed for me
[17:52:14] <paladox>	 ok
[17:52:54] <Reception123>	 "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later." heh yeah getting htis
[17:52:55] <Reception123>	 *this
[17:53:00] <paladox>	 yeh
[17:53:03] <paladox>	 that's ES
[17:53:10] <paladox>	 it hasen't been allocated a shard..
[17:55:24] <Not-7290>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[17:55:25] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjXvP
[17:55:27] <Not-7290>	 [02miraheze/mw-config] 07paladox 03d9a1de8 - Do not merge
[17:55:28] <Not-7290>	 [02mw-config] 07paladox opened pull request 03#2695: Do not merge - 13https://git.io/fjXvX
[17:55:40] <Not-7290>	 [02mw-config] 07paladox synchronize pull request 03#2695: Do not merge - 13https://git.io/fjXvX
[17:55:41] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fjXv1
[17:55:43] <Not-7290>	 [02miraheze/mw-config] 07paladox 03414af59 - Update Redis.php
[18:02:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.52, 7.29, 7.09
[18:03:12] <Reception123>	 !log hack Redis.php on test1 for import
[18:03:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:03:50] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wackypediawiki --report=1 /home/reception/illogicopedia_pages_full.xml on test1 (due to Redis hack)
[18:03:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:08:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.95, 7.90, 7.41
[18:10:07] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 10.04, 8.64, 7.73
[18:23:06] <RhinosF1>	 https://phabricator.miraheze.org/T4536
[18:23:07] <ZppixBot>	 [ ⚓ T4536 Start Miraheze Mailing Lists ] - phabricator.miraheze.org
[18:23:17] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXft
[18:23:19] <Not-7290>	 [02miraheze/puppet] 07paladox 03bf9f065 - elasticsearch: Increase MAX_OPEN_FILES to 150000
[18:23:20] <RhinosF1>	 ^^ feedback wanted
[18:30:07] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.89, 7.98, 7.99
[18:34:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.07, 7.67, 7.81
[18:36:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.63, 7.59, 7.76
[18:45:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXf2
[18:45:13] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03442921a - BOT: Updating services config for wikis
[18:46:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.89, 7.50, 7.46
[18:48:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.68, 6.96, 7.26
[18:54:06] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.79, 5.97, 6.69
[19:10:14] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXfh
[19:10:15] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03b6ccdac - BOT: Updating services config for wikis
[19:22:05] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 7.44, 8.05, 7.17
[19:24:05] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.14, 7.41, 7.04
[19:28:04] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.49, 6.61, 6.77
[19:34:55] <paladox>	 !log root@elasticsearch1:/home/paladox# curl -X POST -H 'Content-Type: application/json' localhost:9200/mw_cirrus_metastore_first/settings -d '{"settings": {"index.priority": 10}}'
[19:35:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:50:48] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXJn
[19:50:49] <Not-7290>	 [02miraheze/mw-config] 07paladox 03b8449ab - Tweek CirrusSearch params
[19:55:58] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.13, 6.82, 6.49
[19:57:58] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.67, 6.33, 6.35
[20:12:14] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXJr
[20:12:16] <Not-7290>	 [02miraheze/puppet] 07paladox 037a83437 - jobrunner: Move CirrusSearch jobs to it's own runner
[20:16:33] <icinga-miraheze>	 PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[20:17:05] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXJP
[20:17:06] <Not-7290>	 [02miraheze/puppet] 07paladox 037a3a462 - Fix syntax
[20:18:21] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXJX
[20:18:23] <Not-7290>	 [02miraheze/puppet] 07paladox 039e55d2f - Update jobrunner.json.erb
[20:20:33] <icinga-miraheze>	 RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[20:21:16] <Not-7290>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[20:21:17] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjXJ1
[20:21:19] <Not-7290>	 [02miraheze/puppet] 07paladox 030765bbc - jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together
[20:21:20] <Not-7290>	 [02puppet] 07paladox opened pull request 03#1028: jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together - 13https://git.io/fjXJM
[20:21:44] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjXJD
[20:21:46] <Not-7290>	 [02miraheze/puppet] 07paladox 03919a713 - Update jobrunner.json.erb
[20:21:47] <Not-7290>	 [02puppet] 07paladox synchronize pull request 03#1028: jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together - 13https://git.io/fjXJM
[20:23:35] <Not-7290>	 [02puppet] 07paladox closed pull request 03#1028: jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together - 13https://git.io/fjXJM
[20:23:36] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXJy
[20:23:38] <Not-7290>	 [02miraheze/puppet] 07paladox 03ac93728 - jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together (#1028) * jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together * Update jobrunner.json.erb
[20:23:39] <Not-7290>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[20:23:41] <Not-7290>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[20:34:16] <paladox>	 !log deleted jobrunner jobs for absolutetruthwiki (including wiki specific jobs)
[20:34:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:35:57] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXUc
[20:35:58] <Not-7290>	 [02miraheze/puppet] 07paladox 03489d7c8 - Revert "jobrunner: Merge DataDumpGenerateJob, GUP and SEARCH queue together (#1028)" This reverts commit ac937282a599aa50917fb0ab6458505e4a1769a1.
[21:03:54] <Not-7290>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[21:03:55] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjXUw
[21:03:57] <Not-7290>	 [02miraheze/puppet] 07paladox 0372128e4 - Add fileLockScript script for matomo archive script
[21:03:58] <Not-7290>	 [02puppet] 07paladox opened pull request 03#1029: Add fileLockScript script for matomo archive script - 13https://git.io/fjXUr
[21:04:10] <Not-7290>	 [02puppet] 07paladox synchronize pull request 03#1029: Add fileLockScript script for matomo archive script - 13https://git.io/fjXUr
[21:04:12] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjXUo
[21:04:13] <Not-7290>	 [02miraheze/puppet] 07paladox 034138263 - Update init.pp
[21:04:48] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjXUK
[21:04:50] <Not-7290>	 [02miraheze/puppet] 07paladox 03c789f25 - Create fileLockScript.sh
[21:04:51] <Not-7290>	 [02puppet] 07paladox synchronize pull request 03#1029: Add fileLockScript script for matomo archive script - 13https://git.io/fjXUr
[21:05:18] <Not-7290>	 [02puppet] 07paladox closed pull request 03#1029: Add fileLockScript script for matomo archive script - 13https://git.io/fjXUr
[21:05:20] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjXUi
[21:05:21] <Not-7290>	 [02miraheze/puppet] 07paladox 03176a6e4 - Add fileLockScript script for matomo archive script (#1029) * Add fileLockScript script for matomo archive script * Update init.pp * Create fileLockScript.sh
[21:05:23] <Not-7290>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[21:05:24] <Not-7290>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[21:10:10] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXUS
[21:10:11] <Not-7290>	 [02miraheze/puppet] 07paladox 03fe2b407 - Fix script (matomo)
[21:39:39] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.012 second response time
[21:41:39] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.017 second response time
[21:44:32] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXTs
[21:44:33] <Not-7290>	 [02miraheze/mw-config] 07paladox 039bafa56 - Sitenotice for ElasticSearch Change-Id: If09ba13a2a08684bdfe9b79050b97caf386903ac
[21:46:24] <paladox>	 !log resetting elasticsearch
[21:46:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:49:39] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.006 second response time
[21:51:39] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.225 second response time
[21:52:21] <RhinosF1>	 paladox: thx, still having issues with search though on meta
[21:52:32] <paladox>	 yeh
[21:55:09] <paladox>	 RhinosF1 known, im currently resetting elasticsearch as it was not recovering the way i wanted it to.
[21:56:06] <MH-Discord>	 <RhinosF1> paladox: cool
[22:06:24] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXT4
[22:06:25] <Not-7290>	 [02miraheze/mw-config] 07paladox 0372f928e - Switch wmgUseCirrusSearch off for all wikis apart from a few
[22:10:12] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXT0
[22:10:13] <Not-7290>	 [02miraheze/mw-config] 07paladox 031d976f8 - CirrusSearch: Reduce shards to 2
[22:12:31] <icinga-miraheze>	 PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 342 bytes in 0.021 second response time
[22:13:25] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.88, 5.98, 4.68
[22:15:24] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.62, 6.10, 4.87
[22:16:24] <icinga-miraheze>	 RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 790 bytes in 0.010 second response time
[22:17:31] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjXTo
[22:17:33] <Not-7290>	 [02miraheze/mw-config] 07paladox 03a937ec7 - Update LocalExtensions.php
[22:27:04] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 5.29, 3.15, 1.59
[22:32:20] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[22:34:20] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 1 failures
[22:38:20] <icinga-miraheze>	 PROBLEM - test1 php-fpm on test1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.3'
[22:38:46] <icinga-miraheze>	 PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[22:41:18] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.16, 7.14, 6.28
[22:45:19] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.85, 7.84, 6.77
[22:59:17] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.69, 6.77, 6.78
[23:03:16] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.20, 7.35, 6.99
[23:05:15] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.02, 7.24, 6.99
[23:09:14] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.83, 6.61, 6.79
[23:32:37] <paladox>	 !log reboot elasticsearch1
[23:32:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master