[02:59:26] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[04:39:34] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 49%
[04:42:10] <icinga-miraheze>	 RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 13%
[04:57:48] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d]
[05:02:55] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[05:33:47] <Not-def6>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Je4zR
[05:33:48] <Not-def6>	 [02miraheze/ssl] 07Reception123 03cf4a3c9 - add wiki.candela.digital cert
[05:45:09] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4za
[05:45:10] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03ec4dc83 - BOT: Updating services config for wikis
[05:47:19] <Not-def6>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Je4zV
[05:47:21] <Not-def6>	 [02miraheze/ssl] 07Reception123 03cb2bd50 - add wiki.itsuali.tk cert
[06:27:19] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3281 MB (13% inode=94%);
[07:15:08] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4gW
[07:15:10] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03966d6b4 - BOT: Updating services config for wikis
[09:40:15] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4a4
[09:40:17] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03071f33c - BOT: Updating services config for wikis
[10:50:48] <SPF|Cloud>	 !log ran importDump.php, rebuildrecentchanges.php and initSiteStats.php for T4776
[10:50:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:17:09] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.01, 6.51, 5.82
[13:19:27] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.64, 5.92, 5.70
[15:23:04] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 12.38, 9.28, 7.13
[15:28:26] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.64, 7.62, 7.14
[15:29:04] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki agoldenbraidwiki /home/reception/delbackups/agoldenbraidwiki.xml
[15:29:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:43:45] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 13.29, 9.32, 7.83
[15:45:25] <Reception123>	 !log deleted caeruleawiki
[15:45:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:45:50] <Reception123>	 !log MariaDB [(none)]> DROP DATABASE caeruleawiki; on db4
[15:45:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:48:36] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.90, 7.94, 7.03
[15:51:01] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.41, 7.45, 6.98
[15:55:41] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.34, 6.69, 6.79
[15:56:11] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 4.75, 7.19, 7.95
[16:01:04] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.73, 5.02, 6.79
[16:24:18] <RhinosF1>	 .at 17:32 meal
[16:24:19] <ZppixBot>	 RhinosF1: Okay, I will set the reminder for: 2019-10-17 - 16:31:59UTC
[16:24:35] <RhinosF1>	 Thx bot
[16:30:14] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki darkfallnewdawnwiki /home/reception/delbackups/darkfallnewdawnwiki.xml
[16:30:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:32:00] <ZppixBot>	 RhinosF1: meal
[16:59:55] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je41I
[16:59:57] <Not-def6>	 [02miraheze/mw-config] 07paladox 039b28413 - remove metawiki and att from new mount
[17:01:49] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je41m
[17:01:51] <Not-def6>	 [02miraheze/mw-config] 07paladox 032841ad8 - Remove new mont
[17:05:34] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 12.85, 8.59, 6.25
[17:08:07] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je413
[17:08:07] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.66, 7.67, 6.27
[17:08:08] <Not-def6>	 [02miraheze/puppet] 07paladox 0320b328f - Update mediawiki.pp
[17:08:25] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je41s
[17:08:27] <Not-def6>	 [02miraheze/puppet] 07paladox 03f0d1489 - Update mediawiki.conf
[17:10:40] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.51, 6.64, 6.08
[17:11:18] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je41W
[17:11:19] <Not-def6>	 [02miraheze/puppet] 07paladox 039827389 - Update mediawiki.pp
[17:16:10] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.59, 7.89, 6.73
[17:18:40] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.70, 7.43, 6.73
[17:21:15] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.73, 6.59, 6.50
[17:21:28] <Not-def6>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[17:21:29] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je41w
[17:21:31] <Not-def6>	 [02miraheze/puppet] 07paladox 038b3857f - mediawiki: Set nginx::worker_processes to auto
[17:21:32] <Not-def6>	 [02puppet] 07paladox opened pull request 03#1107: mediawiki: Set nginx::worker_processes to auto - 13https://git.io/Je41r
[17:21:50] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je41o
[17:21:52] <Not-def6>	 [02miraheze/puppet] 07paladox 035928dd4 - Update mw2.yaml
[17:21:53] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1107: mediawiki: Set nginx::worker_processes to auto - 13https://git.io/Je41r
[17:22:02] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je416
[17:22:03] <Not-def6>	 [02miraheze/puppet] 07paladox 032a2e301 - Update mw3.yaml
[17:22:05] <Not-def6>	 [02puppet] 07paladox synchronize pull request 03#1107: mediawiki: Set nginx::worker_processes to auto - 13https://git.io/Je41r
[17:22:38] <Not-def6>	 [02miraheze/mediawiki] 07Pix1234 pushed 032 commits to 03REL1_33 [+1/-0/±2] 13https://git.io/Je41i
[17:22:40] <Not-def6>	 [02miraheze/mediawiki] 07Pix1234 030b1947a - add CreatePageUw
[17:22:41] <Not-def6>	 [02miraheze/mediawiki] 07Pix1234 032e3cb2b - Merge branch 'REL1_33' of https://github.com/miraheze/mediawiki into REL1_33
[17:22:43] <ZppixBot>	 [ GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. ] - github.com
[17:22:47] <Not-def6>	 [02puppet] 07paladox closed pull request 03#1107: mediawiki: Set nginx::worker_processes to auto - 13https://git.io/Je41r
[17:22:48] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Je41P
[17:22:50] <Not-def6>	 [02miraheze/puppet] 07paladox 033f1e9e3 - mediawiki: Set nginx::worker_processes to auto (#1107) * mediawiki: Set nginx::worker_processes to auto * Update mw2.yaml * Update mw3.yaml
[17:22:51] <Not-def6>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[17:22:53] <Not-def6>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[17:23:18] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 pushed 032 commits to 03master [+0/-0/±5] 13https://git.io/Je411
[17:23:19] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 038acd08d - Test CreatePageUw per T4793
[17:23:21] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 03b8704dd - Merge branch 'master' of https://github.com/miraheze/mw-config
[17:23:22] <ZppixBot>	 [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com
[17:24:30] <travis-ci>	 miraheze/mw-config/master/b8704dd - zppix1 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/599264081
[17:25:03] <paladox>	 ZppixBot you missed a ]
[17:26:39] <paladox>	 ZppixBot you'll need to fix thst ASAP
[17:26:44] <paladox>	 or the sites are going to break
[17:27:08] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.28, 7.53, 6.61
[17:27:49] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je41Q
[17:27:50] <Not-def6>	 [02miraheze/mw-config] 07paladox 03ce71698 - fix
[17:28:19] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Je415
[17:28:21] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 03f8b0352 - syntax
[17:28:22] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 036007463 - Merge branch 'master' of https://github.com/miraheze/mw-config
[17:28:24] <ZppixBot>	 [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com
[17:28:36] <Zppix>	 paladox:  i caught that
[17:28:57] <travis-ci>	 miraheze/mw-config/master/ce71698 - paladox The build is still failing. https://travis-ci.org/miraheze/mw-config/builds/599265789
[17:29:36] <travis-ci>	 miraheze/mw-config/master/6007463 - zppix1 The build is still failing. https://travis-ci.org/miraheze/mw-config/builds/599266057
[17:29:41] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 15.08, 11.91, 8.73
[17:29:52] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je41b
[17:29:54] <Not-def6>	 [02miraheze/mw-config] 07paladox 03ad20d40 - fix
[17:30:18] <Zppix>	 paladox:  syntax is hard ...
[17:30:24] <paladox>	 heh
[17:30:47] <Zppix>	 paladox:  for some reason sublime thinks all is okay but it lies
[17:30:53] <paladox>	 ok
[17:30:58] <travis-ci>	 miraheze/mw-config/master/ad20d40 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/599266655
[17:32:29] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:32:59] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static]
[17:33:13] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.78, 7.85, 5.78
[17:33:15] <Zppix>	 probably my fault ^
[17:34:33] <paladox>	 !log reboot misc4
[17:34:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:35:24] <Zppix>	 !log rebuild lc and mergemessagefiles on test1
[17:35:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:35:36] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:36:29] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 6.19, 6.60, 5.64
[17:38:38] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.43, 7.95, 7.58
[17:43:48] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.79, 6.94, 7.91
[17:43:56] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4Ms
[17:43:57] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 03e6c1015 - fix
[17:44:08] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:44:35] <Reception123>	 Zppix: https://github.com/miraheze/mw-config/blob/master/LocalExtensions.php#L216 if there's no extension.json you need require_once
[17:44:42] <ZppixBot>	 [ mw-config/LocalExtensions.php at master · miraheze/mw-config · GitHub ] - github.com
[17:44:54] <Zppix>	 Reception123:  it says to use WfLoadExt on the ext page
[17:45:26] <Reception123>	 Zppix: but does our branch have an extension.json?
[17:45:37] <Zppix>	 yes
[17:45:59] <Zppix>	 https://github.com/wikimedia/mediawiki-extensions-CreatePageUw/tree/REL1_33
[17:46:00] <ZppixBot>	 [ GitHub - wikimedia/mediawiki-extensions-CreatePageUw at REL1_33 ] - github.com
[17:46:10] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.58, 7.83, 8.11
[17:47:51] <Zppix>	 Reception123:  so i dont get why it errors
[17:48:05] <Reception123>	 Zppix: well if it has an extension.json add it back to extension-list
[17:48:28] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 4.58, 6.47, 7.56
[17:48:41] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4MC
[17:48:42] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 036f7b754 - fix
[17:48:48] <Reception123>	 paladox: around?
[17:48:53] <paladox>	 yup?
[17:49:44] <Reception123>	 paladox: can you check why the extension Zppix is trying to install isn't working?
[17:49:53] <paladox>	 What's the issue?
[17:49:55] <Reception123>	 see -staff, it has an extension.json but the error log says it doesn't
[17:50:33] <paladox>	 root@test1:/var/log/mediawiki/debuglogs# ls -lah /srv/mediawiki/w/extensions/CreatePageUw/
[17:50:33] <paladox>	 total 16K
[17:50:33] <paladox>	 drwxr-xr-x   2 www-data www-data 4.0K Oct 17 17:24 .
[17:50:33] <paladox>	 drwxr-xr-x 228 www-data www-data  12K Oct 17 17:24 ..
[17:50:40] <paladox>	 no one init the module?
[17:50:43] <Reception123>	 Zppix: 
[17:50:44] <Reception123>	 reception@test1:/srv/mediawiki/w/extensions/CreatePageUw$ ls
[17:50:51] <Reception123>	 yeah, found the same thing as paladox
[17:50:53] <Reception123>	 the thing is empty
[17:50:56] <paladox>	 now it works
[17:51:20] <Reception123>	 paladox: why was it empty?
[17:51:31] <Zppix>	 nowhere on the tech: page says anything about initing
[17:51:34] <paladox>	 I guess someone pulled it without running init?
[17:51:51] <paladox>	 you shouldn't have had to
[17:51:53] <Reception123>	 Zppix: normally there's no need
[17:51:55] <paladox>	 puppet should have done it...
[17:52:02] <Reception123>	 I've never inited anything 
[17:52:11] <paladox>	 Did someone git pull?
[17:52:27] <Zppix>	 i was told that i could use either git pull or wait for puppet paladox
[17:52:30] <Reception123>	 not me
[17:52:33] <paladox>	 oh
[17:52:35] <paladox>	 that's why
[17:52:38] <Reception123>	 oh yeah, you shouldn't use git pull
[17:52:42] <Reception123>	 wait for puppet
[17:52:45] <Zppix>	 RhinosF1:  ^
[17:54:36] <Zppix>	 Reception123:  paladox  yay now i can rebuild and stuff again...
[17:54:43] <Zppix>	 !log rebuildlc and mergemessagefiles on test1
[17:54:43] <paladox>	 :)
[17:54:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:00:07] <Zppix>	 !log mergemessagefiles and rebuild lc on mw[123]
[18:00:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:00:21] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je4Mu
[18:00:22] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 03c84f604 - move createpageuw to prod
[18:01:25] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.74, 7.02, 7.06
[18:03:06] <apap>	 quick question, how do i add different types of page protection on my wiki
[18:03:13] <apap>	 for a specific group?
[18:03:23] <Zppix>	 very carefully xD
[18:04:05] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.31, 7.58, 7.17
[18:04:05] <apap>	 ah yes very helpful
[18:04:39] <Zppix>	 apap:  sorry i couldnt resist, i dont think you can do that via managewiki (i may be wrong)
[18:04:47] <Zppix>	 paladox:  ^?
[18:04:48] <apap>	 sad
[18:04:52] <apap>	 oh well
[18:04:55] <paladox>	 Zppix ?
[18:04:57] <apap>	 thanks anyways :)
[18:05:31] <Zppix>	 paladox:  (re apap's question)
[18:05:42] <paladox>	 Hmm
[18:05:52] <paladox>	 I'm not sure
[18:05:53] <paladox>	 Reception123 ^
[18:06:06] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.09, 6.93, 5.85
[18:06:26] * Zppix waits for Reception123  to delegate to the next sysadmin 
[18:06:55] <apap>	 im next sysadmin ;)
[18:07:02] <Not-def6>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Je4Mo
[18:07:04] <Not-def6>	 [02miraheze/MirahezeMagic] 07translatewiki 03ce08455 - Localisation updates from https://translatewiki.net.
[18:07:10] <ZppixBot>	 Hello bounce26! If you have any questions, feel free to ask and someone should answer soon.
[18:07:11] <ZppixBot>	 [ Main page - translatewiki.net ] - translatewiki.net.
[18:08:23] <Zppix>	 apap:  your not a sysadmin
[18:08:34] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.76, 6.54, 5.86
[18:09:15] <apap>	 i wish...
[18:09:35] <apap>	 unless...? :flushed:
[18:09:56] <Zppix>	 ... that was weird
[18:19:42] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.67, 7.52, 7.65
[18:24:35] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.03, 7.73, 7.68
[18:24:53] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.55, 6.65, 7.71
[18:26:57] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.84, 7.30, 7.52
[18:31:49] <ZppixBot>	 Hello Badmuts! If you have any questions, feel free to ask and someone should answer soon.
[18:32:03] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.00, 5.34, 6.66
[18:34:20] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.42, 7.47, 7.44
[18:35:58] <Badmuts>	 Hi! I have a question regarding the country data template. I want to add some flags and flagicons into my wiki, but it seems that there is an error in the str right template or something. I keep having the Expression error: Unexpected < operator. instead of the Template. Can somebody help me out?
[18:36:47] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.13, 7.61, 7.51
[18:51:33] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.74, 5.77, 6.61
[19:11:13] <Badmuts>	 Nobody?
[19:19:31] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.20, 7.24, 5.28
[19:22:05] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.55, 6.52, 5.31
[19:31:45] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.82, 8.33, 6.58
[19:35:50] <RhinosF1>	 .tell apap if you need a new protection level file a Phabricator task with more information
[19:35:51] <ZppixBot>	 RhinosF1: I'll pass that on when apap is around.
[19:39:44] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 4.63, 7.48, 7.19
[19:42:26] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 2.84, 5.65, 6.55
[19:53:42] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.37, 7.78, 6.73
[19:56:04] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.28, 7.00, 6.58
[19:58:28] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.70, 6.16, 6.29
[20:15:34] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.80, 6.93, 6.55
[20:17:55] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.92, 6.25, 6.35
[20:24:05] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.29, 6.67, 6.48
[20:25:38] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:25:43] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:25:48] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 22 failures. Last run 4 minutes ago with 22 failures. Failed resources (up to 3 shown)
[20:25:49] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:25:55] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 4 minutes ago with 15 failures. Failed resources (up to 3 shown)
[20:26:03] <icinga-miraheze>	 PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 4 minutes ago with 15 failures. Failed resources (up to 3 shown)
[20:26:13] <icinga-miraheze>	 PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 4 minutes ago with 16 failures. Failed resources (up to 3 shown)
[20:26:36] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:26:43] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 206 failures. Last run 5 minutes ago with 206 failures. Failed resources (up to 3 shown)
[20:26:54] <icinga-miraheze>	 PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 5 minutes ago with 14 failures. Failed resources (up to 3 shown)
[20:27:12] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 203 failures. Last run 5 minutes ago with 203 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades]
[20:27:12] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.56, 5.70, 6.14
[20:27:13] <icinga-miraheze>	 PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:27:21] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 216 failures. Last run 5 minutes ago with 216 failures. Failed resources (up to 3 shown)
[20:27:26] <icinga-miraheze>	 PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 5 minutes ago with 14 failures. Failed resources (up to 3 shown)
[20:27:52] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 6 minutes ago with 18 failures. Failed resources (up to 3 shown)
[20:28:08] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 6 minutes ago with 51 failures. Failed resources (up to 3 shown)
[20:28:09] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 6 minutes ago with 32 failures. Failed resources (up to 3 shown)
[20:28:26] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 219 failures. Last run 6 minutes ago with 219 failures. Failed resources (up to 3 shown)
[20:28:59] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 163 failures. Last run 6 minutes ago with 163 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades]
[20:33:54] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[20:34:28] <Zppix>	 oh phew i thought i forgot to log what i did on mw hosts
[20:34:36] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[20:34:37] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[20:34:38] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:34:41] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:34:49] <icinga-miraheze>	 RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:34:56] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:35:10] <paladox>	 Zppix well if you did, your definitly not going to cause that issue :P
[20:35:29] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:35:33] <icinga-miraheze>	 RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:35:51] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[20:35:52] <icinga-miraheze>	 RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:35:56] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:36:01] <icinga-miraheze>	 RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:36:15] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:36:37] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:36:54] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:37:10] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:37:32] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:37:49] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:39:15] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.45, 8.90, 7.43
[20:44:27] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.22, 7.77, 7.38
[20:56:56] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.17, 6.06, 6.74
[21:05:06] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.93, 7.55, 7.16
[21:11:35] <RhinosF1>	 Zppix: time to hide! Your deployment train window has just pulled up!
[21:12:32] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.64, 6.01, 6.67
[21:15:28] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.87, 7.45, 5.67
[21:22:31] <Not-def6>	 [02miraheze/mediawiki] 07Pix1234 pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/Je49D
[21:22:33] <Not-def6>	 [02miraheze/mediawiki] 07Pix1234 03534f0a8 - add RecentChangesLogFilter ext
[21:22:57] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Je49y
[21:22:59] <Not-def6>	 [02miraheze/mw-config] 07Pix1234 032235d34 - testing RecentChangesLogFilter ext
[21:25:08] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.83, 8.30, 6.78
[21:25:09] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je49H
[21:25:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03bb2ec9e - BOT: Updating services config for wikis
[21:27:37] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.21, 7.90, 6.86
[21:28:57] <Zppix>	 !log rebuild lc and mergemessagefiles on test1
[21:29:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:33:01] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.06, 7.78, 7.05
[21:33:23] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:35:41] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.18, 7.33, 7.02
[21:35:54] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[21:38:18] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.37, 7.89, 7.23
[21:46:03] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.12, 6.01, 6.80
[21:57:34] <Zppix>	 !log rebuild rc on test1wiki on test1
[21:57:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:13:22] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.01, 7.13, 6.01
[22:18:25] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.97, 7.89, 6.72
[22:23:27] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.86, 7.68, 6.82
[22:25:56] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.33, 6.58, 6.53
[22:29:08] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.94, 7.41, 6.57
[22:31:40] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.77, 8.26, 7.02
[22:35:26] <JohnLewis>	 !log  UPDATE actor SET actor_name = 'Flow talk page manager2' WHERE actor_id = 1471; on oecumenewiki
[22:35:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:39:17] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.85, 7.69, 7.31
[22:40:20] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.82, 6.74, 6.71
[22:42:48] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.67, 5.78, 6.34
[22:46:47] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 9.17, 7.31, 7.14
[22:49:05] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.96, 7.40, 7.22
[22:56:17] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.82, 6.00, 6.64
[23:03:31] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.19, 7.01, 6.04
[23:04:34] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.61, 7.39, 7.04
[23:05:55] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.65, 5.94, 5.78
[23:09:35] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.16, 6.64, 6.80
[23:13:07] <Not-def6>	 [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3
[23:13:09] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Je4Qd
[23:13:10] <Not-def6>	 [02miraheze/mw-config] 07paladox 033993ac6 - Remove commons.wikimedia.org cache This will improve performance and also use less disk space. This is because it won't be generating thumberneils (and can fetch from commons which is faster). It'll also use less disk space as it won't be saving to disk. And lastly it'll use less bandwith for the mediawiki servers as the client will be fetching directly.
[23:13:12] <Not-def6>	 [02mw-config] 07paladox opened pull request 03#2777: Remove commons.wikimedia.org cache - 13https://git.io/Je4QF
[23:14:23] <Not-def6>	 [02mw-config] 07paladox closed pull request 03#2777: Remove commons.wikimedia.org cache - 13https://git.io/Je4QF
[23:14:24] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je4QA
[23:14:26] <Not-def6>	 [02miraheze/mw-config] 07paladox 034f24f47 - Remove commons.wikimedia.org cache (#2777) This will improve performance and also use less disk space. This is because it won't be generating thumberneils (and can fetch from commons which is faster). It'll also use less disk space as it won't be saving to disk. And lastly it'll use less bandwith for the mediawiki servers as the client will be fetching directly.
[23:14:27] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3
[23:14:29] <Not-def6>	 [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3
[23:15:12] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.43, 7.31, 7.03
[23:22:37] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.42, 6.31, 6.69
[23:57:03] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je47M
[23:57:04] <Not-def6>	 [02miraheze/puppet] 07paladox 03bf8b541 - Update mediawiki.pp
[23:58:09] <paladox>	 !log depool and repool mw[123]
[23:58:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master