[00:37:23] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%);
[00:41:46] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBPP
[00:41:47] <Not-def6>	 [02miraheze/puppet] 07paladox 03ed5042a - Update misc1.yaml
[02:32:48] <Not-def6>	 [02puppet] 07The-Voidwalker opened pull request 03#1109: use internal mail - 13https://git.io/JeB1Y
[02:33:40] <Not-def6>	 [02puppet] 07paladox closed pull request 03#1109: use internal mail - 13https://git.io/JeB1Y
[02:33:42] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB13
[02:33:43] <Not-def6>	 [02miraheze/puppet] 07The-Voidwalker 033598300 - use internal mail (#1109)
[04:15:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB1j
[04:15:13] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 034fd0f09 - BOT: Updating services config for wikis
[04:26:49] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.22, 6.32, 4.99
[04:28:52] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 10.89, 8.16, 5.83
[04:30:37] <icinga-miraheze>	 PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 51203 MB (10% inode=99%);
[04:47:10] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.06, 7.76, 7.62
[04:51:15] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.91, 5.70, 6.79
[04:53:46] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.79, 7.09, 5.99
[04:55:47] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.01, 7.57, 6.30
[04:57:48] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.47, 6.90, 6.21
[04:59:48] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.44, 6.49, 6.15
[06:04:23] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis]
[06:12:23] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[06:25:57] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3200 MB (13% inode=94%);
[06:39:36] <icinga-miraheze>	 PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Tue 05 Nov 2019 06:37:14 AM GMT +0000).
[06:39:50] <Not-def6>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBMx
[06:39:51] <Not-def6>	 [02miraheze/ssl] 07MirahezeSSLBot 03d754598 - Bot: Update SSL cert for wiki.counterculturelabs.org
[06:43:38] <icinga-miraheze>	 RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Sat 18 Jan 2020 05:39:44 AM GMT +0000.
[07:02:28] <icinga-miraheze>	 PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: DISK CRITICAL - free space: / 28471 MB (5% inode=99%);
[08:30:12] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBDQ
[08:30:14] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03544f2a4 - BOT: Updating services config for wikis
[09:28:04] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.80, 7.15, 5.58
[09:34:04] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.56, 7.79, 6.39
[09:38:04] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.44, 7.64, 6.61
[09:40:05] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.38, 7.34, 6.61
[09:44:04] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.14, 6.68, 6.50
[09:52:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.50, 7.25, 6.74
[09:54:01] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.84, 7.46, 6.87
[10:02:03] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.25, 6.72, 6.71
[10:05:00] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.23, 6.63, 6.12
[10:06:07] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.40, 7.03, 6.83
[10:07:00] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 6.10, 6.58, 6.17
[10:16:10] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.64, 7.27, 6.71
[10:18:01] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.53, 5.65, 6.52
[10:20:10] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.32, 7.45, 6.90
[10:22:09] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.17, 7.43, 6.97
[10:32:08] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.35, 5.98, 6.55
[12:10:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB9Y
[12:10:13] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 030c42b44 - BOT: Updating services config for wikis
[12:56:54] <ZppixBot>	 Hello Inven! If you have any questions, feel free to ask and someone should answer soon.
[12:57:13] <RhinosF1>	 Welcome
[12:57:50] <Inven>	 This is the first time I use IRC
[12:58:18] <RhinosF1>	 We're happy to help
[12:59:32] <paladox>	 !log reboot misc4
[12:59:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:07:46] <paladox>	 !log root@misc4:/srv/parsoid# npm install node-worker-farm@1.5.4
[13:07:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:08:59] <paladox>	 That fixes a issue with parsoid not killing it's workers...
[13:09:40] <Inven>	 !helllp
[13:09:46] <Inven>	 !help
[13:11:22] <paladox>	 !log upgrade phabricator on misc4
[13:11:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:11:43] <paladox>	 Inven hi, how can we help you?
[13:12:20] <Inven>	 No thanks
[13:18:38] <paladox>	 !log root@misc4:/srv/parsoid# npm install worker-farm@1.7.0
[13:18:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:25:45] <icinga-miraheze>	 PROBLEM - misc4 parsoid on misc4 is CRITICAL: connect to address 185.52.3.121 and port 8142: Connection refused
[13:27:40] <RhinosF1>	 paladox: is that expected?
[13:27:47] <paladox>	 yes
[13:27:54] <RhinosF1>	 K
[13:28:10] <paladox>	 !log root@misc4:/srv/parsoid# sudo -u root npm install service-runner@2.6.19
[13:28:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:29:43] <icinga-miraheze>	 RECOVERY - misc4 parsoid on misc4 is OK: TCP OK - 0.001 second response time on 185.52.3.121 port 8142
[13:37:50] <icinga-miraheze>	 PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures
[13:43:46] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.09, 7.31, 5.32
[13:45:46] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.77, 7.08, 5.48
[13:47:30] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBQq
[13:47:31] <Not-def6>	 [02miraheze/puppet] 07paladox 03d8a3b0c - services: Tweek parsoid config
[13:47:48] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.59, 8.45, 6.18
[13:49:48] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.13, 7.15, 5.97
[13:51:50] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.69, 6.40, 5.85
[13:55:22] <Not-def6>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/JeBQn
[13:55:24] <Not-def6>	 [02miraheze/mediawiki] 07paladox 037773068 - Add ParsoidBatchAPI MW extension
[13:56:19] <Not-def6>	 [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3
[13:56:21] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JeBQc
[13:56:22] <Not-def6>	 [02miraheze/mw-config] 07paladox 0356e191d - Install ParsoidBatchAPI as a global extension
[13:56:24] <Not-def6>	 [02mw-config] 07paladox opened pull request 03#2779: Install ParsoidBatchAPI as a global extension - 13https://git.io/JeBQC
[13:57:00] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JeBQ8
[13:57:02] <Not-def6>	 [02miraheze/mw-config] 07paladox 037b14cd3 - Update extension-list
[13:57:03] <Not-def6>	 [02mw-config] 07paladox synchronize pull request 03#2779: Install ParsoidBatchAPI as a global extension - 13https://git.io/JeBQC
[13:57:42] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.79, 6.87, 6.50
[13:58:49] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 6.59, 6.85, 6.26
[13:59:10] <Not-def6>	 [02mw-config] 07paladox closed pull request 03#2779: Install ParsoidBatchAPI as a global extension - 13https://git.io/JeBQC
[13:59:11] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeBQB
[13:59:13] <Not-def6>	 [02miraheze/mw-config] 07paladox 03d391806 - Install ParsoidBatchAPI as a global extension (#2779) * Install ParsoidBatchAPI as a global extension * Update extension-list
[13:59:14] <Not-def6>	 [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3
[13:59:16] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3
[13:59:41] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.52, 6.64, 6.45
[14:00:48] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 11.25, 8.18, 6.80
[14:00:56] <paladox>	 !log running lc on mw[123]
[14:01:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:02:59] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 11.03, 7.22, 5.80
[14:03:38] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.04, 7.39, 6.78
[14:04:13] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBQg
[14:04:14] <Not-def6>	 [02miraheze/puppet] 07paladox 03214d498 - Update config.yaml
[14:06:11] <icinga-miraheze>	 RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[14:06:58] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.82, 7.81, 6.40
[14:07:31] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.89, 7.51, 6.99
[14:08:58] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.33, 8.34, 6.77
[14:10:57] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 5.69, 7.46, 6.65
[14:12:56] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.74, 6.75, 6.47
[14:14:10] <RhinosF1>	 paladox: what’s that going to do?
[14:14:37] <paladox>	 That's going to reduce cpu cycles when using parsoid https://www.mediawiki.org/wiki/Extension:ParsoidBatchAPI
[14:14:38] <ZppixBot>	 [ Extension:ParsoidBatchAPI - MediaWiki ] - www.mediawiki.org
[14:18:52] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.88, 6.95, 7.80
[14:23:46] <Not-def6>	 [02miraheze/mediawiki] 07paladox pushed 0337 commits to 03REL1_33 [+76/-4/±81] 13https://git.io/JeBQA
[14:23:47] <Not-def6>	 [02miraheze/mediawiki] 07paladox 03a1742c4 - Fix mistake in release notes Change-Id: I0eb86a1e4d2b6a5e810231ffd6081295dff18b58
[14:23:49] <Not-def6>	 [02miraheze/mediawiki] 07reedy 03c40c57f - Add 1.33.1 section to RELEASE-NOTES Change-Id: Ie276cab8dd0fafd19c361349438bc0a0df58304d
[14:23:50] <Not-def6>	 [02miraheze/mediawiki] 07zaori 0317f377d - Update git submodules * Update skins/Timeless from branch 'REL1_33' to 3980352ef0657a1a0230e77624d2e7015a9d0c86 - New font stack, plus some generic forms and crap Follow-up on https://gerrit.wikimedia.org/r/501850: * new font stack stolen from phabricator and then rearranged a bit to be less ornery * restored searchinput fonts and prodded some other alignment issues *
[14:23:51] <Not-def6>	 p-personal alignment issues maybe addressed, maybe made worse; cannot test windows/mac because don't have either of them or any remaining budget with which to purchase either of them because holy crap expensive * some very basic generic form styles to align the base ones a tad more consistently * dumb fix for dateinput widgets being mixed into otherwise non-ooui forms Bug: T221345 Change-Id:
[14:23:51] <Not-def6>	 I43312b23197ed1ce5ea9cb75fb0b8e8466bac6a1 (cherry picked from commit 2a4fb34b630b1da704243d981b478951d26df1a5)
[14:23:52] <Not-def6>	 [02miraheze/mediawiki] ... and 34 more commits.
[14:24:58] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.82, 5.28, 6.76
[14:25:47] <RhinosF1>	 paladox: while you're on the repos can you check if M/CW are up to date with i18n
[14:27:09] <paladox>	 done
[14:27:17] <Not-def6>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±3] 13https://git.io/JeBQh
[14:27:18] <Not-def6>	 [02miraheze/mediawiki] 07paladox 03a88f2d9 - Update CW, MW and MM
[14:27:38] <RhinosF1>	 :)
[14:27:51] <RhinosF1>	 don't forget rebuild lc etc etc
[14:29:48] <paladox>	 !log run lc on mw[123], test1
[14:29:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:44:41] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.70, 6.56, 6.76
[14:52:29] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.91, 7.00, 6.85
[14:55:10] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB7F
[14:55:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03bcb2792 - BOT: Updating services config for wikis
[14:58:14] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.76, 7.95, 7.26
[15:02:06] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.14, 7.26, 7.15
[15:08:03] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.58, 6.43, 6.77
[15:19:57] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 12.47, 9.38, 6.78
[15:21:39] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB5l
[15:21:41] <Not-def6>	 [02miraheze/puppet] 07paladox 03cd0b309 - Update config.yaml
[15:22:00] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.28, 7.97, 6.61
[15:23:14] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeB58
[15:23:15] <Not-def6>	 [02miraheze/puppet] 07paladox 032c7a81e - Update config.yaml
[15:26:02] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 3.80, 6.06, 6.15
[16:18:43] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBd2
[16:18:45] <Not-def6>	 [02miraheze/puppet] 07paladox 036e90dec - Update mfschunkserver.cfg.erb
[16:20:29] <paladox>	 !log restart lizardfs-chunkserver on lizardfs[45]
[16:20:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:20:55] <RhinosF1>	 revi: can I get a favour? https://phabricator.miraheze.org/T4828
[16:20:56] <ZppixBot>	 [ ⚓ T4828 naruhito.miraheze.orgでTitleBlacklistを有効にしてください ] - phabricator.miraheze.org
[16:24:11] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[16:24:12] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:24:29] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:24:46] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[16:24:52] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:25:18] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:25:19] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[16:25:46] <RhinosF1>	 paladox: ^
[16:25:54] <paladox>	 it's back
[16:26:06] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[16:26:07] <RhinosF1>	 paladox: good
[16:26:10] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[16:26:10] <paladox>	 that was me when i restarted the chunkserver
[16:26:16] <RhinosF1>	 paladox: oh
[16:26:22] <RhinosF1>	 What's our uptime?
[16:26:29] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.684 second response time
[16:26:44] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[16:26:48] <paladox>	 RhinosF1 ?
[16:26:51] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.692 second response time
[16:27:12] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:27:18] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[16:27:44] <RhinosF1>	 paladox: how long were we up for continuosly?
[16:27:52] <paladox>	 i'm not sure
[16:27:59] <RhinosF1>	 paladox: oh
[16:28:09] <RhinosF1>	 Did we ever reach 50 hours?
[16:28:13] <paladox>	 No
[16:28:23] <paladox>	 because we had a blip yesturday
[16:28:27] <RhinosF1>	 paladox: what did we get?
[16:28:36] <paladox>	 I'm not sure....
[16:28:42] <revi>	 RhinosF1: wrong language
[16:28:52] <revi>	 It is Japanese
[16:29:12] <revi>	 and I am ja-0, go find The_Pioneer
[16:29:24] <RhinosF1>	 revi: will do
[16:29:37] <RhinosF1>	 paladox: where do you get the stats from?
[16:30:09] <paladox>	 grafana.miraheze.org
[16:30:54] <RhinosF1>	 paladox: which view? Link?
[16:31:44] <paladox>	 that link :)
[16:31:52] <paladox>	 well i look at the cpu usage
[16:32:03] <RhinosF1>	 paladox: ah
[16:32:08] <paladox>	 if it decreases significantly then that means the mw is out of service.
[16:32:12] <RhinosF1>	 Uptime on that is misleading
[16:33:26] <paladox>	 no?
[16:33:38] <RhinosF1>	 paladox: there is actual uptime sections
[16:33:46] <RhinosF1>	 They say XX weeks
[16:33:49] <paladox>	 Huh?
[16:33:50] <paladox>	 Yes
[16:33:53] <paladox>	 that's server uptime
[16:34:32] <RhinosF1>	 paladox: k
[16:34:44] <RhinosF1>	 Tell me when I owe you a barnstar
[16:35:58] <RhinosF1>	 revi: there is a ko wiki on https://meta.miraheze.org/wiki/Special:RequestWikiQueue
[16:35:59] <ZppixBot>	 [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org
[16:36:01] <RhinosF1>	 Can you look?
[17:02:12] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBF8
[17:02:13] <Not-def6>	 [02miraheze/puppet] 07paladox 033549857 - Update groups.conf
[17:25:09] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBFQ
[17:25:11] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 03a0dec79 - BOT: Updating services config for wikis
[17:31:28] <RhinosF1>	 apap: please feel free to consider an alternative proposal that could achieve support
[17:31:44] <RhinosF1>	 Hey Voidwalker, just discussing the nightmare RfC I created for you
[17:33:48] <Voidwalker>	 oh boy
[17:34:29] <apap>	 i'm gonna have some trouble with that since working with others who are limited with English or cannot understand English at all will probably have trouble with communicating their troubles..
[17:34:29] <RhinosF1>	 Voidwalker: it's the on wiki version of hell
[17:35:18] <RhinosF1>	 apap: it's complex, I'm going to post a comment in the comments section pinging those neutral/opposed on another way to implement the view on how to define the terms
[17:43:54] <RhinosF1>	 apap: good luck :)
[17:44:49] <apap>	 ty
[17:44:58] <apap>	 :)
[17:52:20] <ZppixBot>	 Hello Egg36! If you have any questions, feel free to ask and someone should answer soon.
[17:53:03] <Egg36>	 hello
[17:54:16] <Voidwalker>	 hi
[17:54:41] <Egg36>	 i am having a hard time understanding the documentation about extensions
[17:56:03] <Voidwalker>	 what do you need help with?
[17:57:36] <Egg36>	 (sorry there is a wasp buzzing around i'm distracted by it)
[17:58:53] <Egg36>	 Do the extensions on this page https://archivetransyahoo.miraheze.org/wiki/Special:ManageWiki/extensions  under "Extension available via Special:ManageWiki" require special accesss to use?
[17:58:54] <ZppixBot>	 [ Manage a wiki - ArchiveTransYahoo ] - archivetransyahoo.miraheze.org
[17:59:11] <Egg36>	 wait
[17:59:14] <Egg36>	 wropng url
[17:59:23] <Egg36>	 oh no
[17:59:29] <Egg36>	 right url
[17:59:55] <Egg36>	 i saw somewhere else you have to send a request via phabricator
[18:00:01] <Egg36>	 and i can't see them on the linked page
[18:00:14] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBbV
[18:00:15] <Not-def6>	 [02miraheze/services] 07MirahezeSSLBot 0380bb813 - BOT: Updating services config for wikis
[18:00:18] <Voidwalker>	 they shouldn't, although some may have a few restrictions
[18:00:32] <RhinosF1>	 Egg36: anyone with the ‘managewiki’ right on a wiki can turn them on unless they are restricted (in which case let me know and I can help)
[18:01:11] <Egg36>	 this is the place where they would be right? https://archivetransyahoo.miraheze.org/wiki/Special:ManageWiki/extensions
[18:01:12] <ZppixBot>	 [ Manage a wiki - ArchiveTransYahoo ] - archivetransyahoo.miraheze.org
[18:03:47] <RhinosF1>	 Egg36: it is
[18:04:21] <Egg36>	 ok thank you
[18:04:45] <Egg36>	 if i wanted several should i ask here or make a request at https://phabricator.miraheze.org/maniphest/task/edit/form/7/ or some other way?
[18:04:46] <ZppixBot>	 [ Login ] - phabricator.miraheze.org
[18:07:29] <RhinosF1>	 Egg36: if you need any not listed or restricted, please create an individual task for each one
[18:07:36] <RhinosF1>	 On phabricator
[18:13:23] <Egg36>	 OK got it
[18:13:23] <Egg36>	 thanks
[18:13:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.08, 6.87, 5.99
[18:15:30] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.21, 6.61, 6.00
[18:16:49] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%);
[18:23:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.30, 7.53, 6.55
[18:27:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.95, 6.96
[18:29:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.97, 8.26, 7.18
[18:46:53] <MH-Discord>	 <RhinosF1> @Stewards PM
[18:47:12] <RhinosF1>	 Voidwalker, JohnLewis ^
[18:47:22] <Voidwalker>	 go ahead
[18:48:33] <RhinosF1>	 {{done}}
[18:50:18] <JohnLewis>	 assume resolved?
[18:52:47] <Voidwalker>	 fairly, still deciding what to do
[18:53:41] <JohnLewis>	 2O?
[19:00:34] <Voidwalker>	 ?
[19:00:53] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.51, 6.87, 5.65
[19:01:42] <JohnLewis>	 Voidwalker: 2(nd)O(pinion) :P
[19:02:10] <Voidwalker>	 oh, pm then?
[19:04:50] <JohnLewis>	 sure
[19:04:56] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.92, 6.72, 5.89
[19:13:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.40, 6.25, 7.91
[19:23:31] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.45, 5.43, 6.66
[19:25:41] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 11.66, 8.95, 6.92
[19:27:39] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 4.61, 7.23, 6.54
[19:29:38] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.42, 6.24, 6.25
[20:51:34] <Not-def6>	 [02mediawiki] 07paladox deleted branch 03REL1_32 - 13https://git.io/vbL5b
[20:51:35] <Not-def6>	 [02miraheze/mediawiki] 07paladox deleted branch 03REL1_32
[20:58:00] <Not-def6>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/JeBx0
[20:58:02] <Not-def6>	 [02miraheze/mediawiki] 07paladox 037bd2485 - Update .gitmodules
[20:59:26] <RhinosF1>	 paladox: what's ignore dirty mean?
[21:00:02] <paladox>	 It means if you dirty the submodule it won't show up as so when running "git status" which slows performance for puppet.
[21:00:28] <RhinosF1>	 ok
[21:24:07] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.88, 6.91, 6.15
[21:26:03] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.54, 6.26, 6.00
[21:41:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.88, 5.89, 5.63
[21:43:30] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.75, 5.72, 5.59
[22:13:28] <Not-def6>	 [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3
[22:13:29] <Not-def6>	 [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JeBhe
[22:13:31] <Not-def6>	 [02miraheze/mw-config] 07RhinosF1 0364c63a1 - see -staff fix due to NGINX 301
[22:13:43] <Not-def6>	 [02mw-config] 07RhinosF1 opened pull request 03#2780: see -staff - 13https://git.io/JeBhf
[22:13:53] <RhinosF1>	 paladox: ^
[22:14:13] <paladox>	 oh, ha, i thought that was commonswiki :P
[22:14:14] <paladox>	 good spot
[22:14:34] <paladox>	 RhinosF1 needs a better commit msg title
[22:14:42] <RhinosF1>	 paladox: doing
[22:15:27] <Not-def6>	 [02mw-config] 07RhinosF1 edited pull request 03#2780: commonswiki.miraheze.org --> commons.miraheze.org - 13https://git.io/JeBhf
[22:15:37] <RhinosF1>	 paladox: ^
[22:15:51] <Not-def6>	 [02mw-config] 07paladox edited pull request 03#2780: Fix description url for commons.miraheze.org - 13https://git.io/JeBhf
[22:15:55] <Not-def6>	 [02mw-config] 07paladox closed pull request 03#2780: Fix description url for commons.miraheze.org - 13https://git.io/JeBhf
[22:15:57] <Not-def6>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBh3
[22:15:58] <Not-def6>	 [02miraheze/mw-config] 07RhinosF1 030739344 - see -staff (#2780) fix due to NGINX 301
[22:16:03] <Not-def6>	 [02mw-config] 07paladox deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3
[22:16:04] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03RhinosF1-patch-3
[22:16:33] <Not-def6>	 [02mw-config] 07paladox deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3
[22:16:34] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03RhinosF1-patch-2
[22:16:36] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03Southparkfan-patch-2
[22:16:38] <Not-def6>	 [02mw-config] 07paladox deleted branch 03Southparkfan-patch-2 - 13https://git.io/vbvb3
[22:16:39] <Not-def6>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[22:16:40] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[22:16:53] <Not-def6>	 [02miraheze/mw-config] 07paladox deleted branch 03RhinosF1-patch-1
[22:16:55] <Not-def6>	 [02mw-config] 07paladox deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3
[22:47:30] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.98, 6.55, 5.87
[22:49:30] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.65, 6.18, 5.81
[23:06:49] <Not-def6>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBjJ
[23:06:50] <Not-def6>	 [02miraheze/puppet] 07JohnFLewis 03d9dca5a - remove policy emails from postfix
[23:07:03] <Not-def6>	 [02puppet] 07Pix1234 opened pull request 03#1110: Add additional ssh key for Zppix - 13https://git.io/JeBjT
[23:08:02] <Not-def6>	 [02puppet] 07paladox reviewed pull request 03#1110 commit - 13https://git.io/JeBjk
[23:08:56] <Not-def6>	 [02puppet] 07Pix1234 synchronize pull request 03#1110: Add additional ssh key for Zppix - 13https://git.io/JeBjT
[23:09:36] <Not-def6>	 [02puppet] 07paladox closed pull request 03#1110: Add additional ssh key for Zppix - 13https://git.io/JeBjT
[23:09:38] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBjL
[23:09:39] <Not-def6>	 [02miraheze/puppet] 07Pix1234 0339444d0 - Add additional ssh key for Zppix (#1110) * Add additional ssh key for Zppix * Spacing
[23:18:45] <Not-def6>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeBjn
[23:18:47] <Not-def6>	 [02miraheze/puppet] 07paladox 039805b1f - puppetserver: Change ensure to present We have a cron that pulls, no need to do duplicate jobs.