[00:03:40] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[00:06:09] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[00:08:16] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[00:09:57] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvff7
[00:09:58] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[00:09:59] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03a82ad27 - Switch from redis to memcached for $wgMainCacheType This reverts commit 335cb477e56cbcb70e1b3cb57636091b44b69c8a.
[01:04:18] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfJd
[01:04:19] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03425c129 - mediawiki: Increase memcache heap to 256M
[01:14:17] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JvfJh
[01:14:18] <Not-5a0a>	 [02miraheze/puppet] 07paladox 030c3c564 - Only set mediawiki::php::use_tideways to true on test1
[01:14:20] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS
[01:14:21] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1179: Only set mediawiki::php::use_tideways to true on test1 - 13https://git.io/JvfJj
[01:15:43] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JvfUJ
[01:15:45] <Not-5a0a>	 [02miraheze/puppet] 07paladox 034919d80 - Update test1.yaml
[01:15:46] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1179: Only set mediawiki::php::use_tideways to true on test1 - 13https://git.io/JvfJj
[01:15:54] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1179: Only set mediawiki::php::use_tideways to true on test1 - 13https://git.io/JvfJj
[01:15:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvfUU
[01:15:57] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03bfb17a2 - Only set mediawiki::php::use_tideways to true on test1 (#1179) * Only set mediawiki::php::use_tideways to true on test1 * Update test1.yaml
[01:15:59] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8
[01:16:00] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS
[01:24:54] <Not-5a0a>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfUq
[01:24:56] <Not-5a0a>	 [02miraheze/mw-config] 07Southparkfan 032921c05 - Use full hostname for test1
[01:37:16] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JvfUC
[01:37:18] <Not-5a0a>	 [02miraheze/puppet] 07paladox 035e86f40 - mediawiki Switch memcached.conf to be an erb template
[01:37:19] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS
[01:37:21] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1180: mediawiki Switch memcached.conf to be an erb template - 13https://git.io/JvfUW
[01:37:34] <Not-5a0a>	 [02puppet] 07paladox edited pull request 03#1180: mediawiki Switch memcached.conf to be an erb template - 13https://git.io/JvfUW
[01:40:02] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+1/-1/±0] 13https://git.io/JvfUB
[01:40:03] <Not-5a0a>	 [02miraheze/puppet] 07paladox 036355d9b - Update and rename modules/mediawiki/files/memcached.conf to modules/mediawiki/templates/memcached.conf.erb
[01:40:05] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1180: mediawiki Switch memcached.conf to be an erb template - 13https://git.io/JvfUW
[01:51:27] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://git.io/JvfUV
[01:51:28] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c7f4c83 - redis: make heap and firewall configurable
[01:51:30] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://git.io/vbiAS
[01:51:31] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1181: redis: make heap and firewall configurable - 13https://git.io/JvfUw
[01:55:28] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JvfUr
[01:55:30] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03feed3e8 - mediawiki: Add a local redis instance configurable by use_redis Also switch on redis and also set heap to 3000mb for lizardfs6
[01:55:31] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS
[01:55:33] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1182: mediawiki: Add a local redis instance configurable by use_redis - 13https://git.io/JvfUo
[01:56:01] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JvfUK
[01:56:02] <Not-5a0a>	 [02miraheze/puppet] 07paladox 038499abd - Update common.yaml
[01:56:04] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1182: mediawiki: Add a local redis instance configurable by use_redis - 13https://git.io/JvfUo
[01:56:54] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JvfU6
[01:56:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c2a0683 - Update lizardfs6.yaml
[01:56:57] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1182: mediawiki: Add a local redis instance configurable by use_redis - 13https://git.io/JvfUo
[01:58:28] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1182: mediawiki: Add a local redis instance configurable by use_redis - 13https://git.io/JvfUo
[01:58:30] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvfUi
[01:58:32] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0318c5c98 - mediawiki: Add a local redis instance configurable by use_redis (#1182) * mediawiki: Add a local redis instance configurable by use_redis Also switch on redis and also set heap to 3000mb for lizardfs6 * Update common.yaml * Update lizardfs6.yaml
[01:58:33] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://git.io/vbiAS
[01:58:35] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10
[02:00:20] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvfUP
[02:00:21] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0384b7804 - redis: Define redis-central and redis-local
[02:00:23] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[02:00:24] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2855: redis: Define redis-central and redis-local - 13https://git.io/JvfUX
[02:02:26] <Not-5a0a>	 [02mw-config] 07paladox synchronize pull request 03#2855: redis: Define redis-central and redis-local - 13https://git.io/JvfUX
[02:02:27] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvfUD
[02:02:29] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0343ce704 - Update Redis.php
[02:03:28] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[redis-server]
[02:03:39] <Not-5a0a>	 [02mw-config] 07paladox closed pull request 03#2855: redis: Define redis-central and redis-local - 13https://git.io/JvfUX
[02:03:40] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfUy
[02:03:42] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03953f7e3 - redis: Define redis-central and redis-local (#2855) * redis: Define redis-central and redis-local * Update Redis.php
[02:03:43] <Not-5a0a>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[02:03:45] <Not-5a0a>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[02:03:46] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[redis-server]
[02:04:39] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[redis-server]
[02:04:47] <travis-ci>	 miraheze/mw-config/paladox-patch-1/43ce704 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/635552115
[02:12:28] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[02:12:28] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[02:12:31] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[02:13:02] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JvfUN
[02:13:04] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03fe16d5d - prometheus: Load all hosts that use redis in yaml file
[02:13:05] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS
[02:13:07] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1183: prometheus: Load all hosts that use redis in yaml file - 13https://git.io/JvfUA
[02:13:48] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/JvfUx
[02:13:49] <Not-5a0a>	 [02miraheze/puppet] 07paladox 033aeb793 - Update init.pp
[02:13:51] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1183: prometheus: Load all hosts that use redis in yaml file - 13https://git.io/JvfUA
[02:14:15] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1183: prometheus: Load all hosts that use redis in yaml file - 13https://git.io/JvfUA
[02:14:16] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvfUh
[02:14:18] <Not-5a0a>	 [02miraheze/puppet] 07paladox 036b170b2 - prometheus: Load all hosts that use redis in yaml file (#1183) * prometheus: Load all hosts that use redis in yaml file * Update init.pp
[02:27:35] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfTL
[02:27:36] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03682d6f3 - prometheus: Update redis exporter to 1.3.x
[02:30:39] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvfTm
[02:30:41] <Not-5a0a>	 [02miraheze/puppet] 07paladox 034069f18 - Revert "prometheus: Load all hosts that use redis in yaml file (#1183)" This reverts commit 6b170b2c24eaa6169e2fb567c8908c148ceecc06.
[02:40:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfTc
[02:40:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03fb5c4b3 - BOT: Updating services config for wikis
[02:45:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfTB
[02:45:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 038247ecb - BOT: Updating services config for wikis
[02:54:50] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfTg
[02:54:51] <Not-5a0a>	 [02miraheze/puppet] 07paladox 037727331 - Update init.pp
[03:17:53] <icinga-miraheze>	 PROBLEM - wiki.opendominion.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.opendominion.net' expires in 15 day(s) (Mon 27 Jan 2020 03:15:17 AM GMT +0000).
[03:18:06] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfTA
[03:18:08] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 034e98996 - Bot: Update SSL cert for wiki.opendominion.net
[03:21:57] <icinga-miraheze>	 RECOVERY - wiki.opendominion.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.opendominion.net' will expire on Fri 10 Apr 2020 02:18:00 AM GMT +0000.
[03:34:12] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvfkq
[03:34:13] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 039937477 - Doin't use redis-local ManageWiki updates the cache when it updates permissions/namespaces so by using redis-local only one server will get the updated version leaving the rest with stale versions.
[04:04:47] <paladox>	 !log depool mw1
[04:04:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:07:26] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1
[04:07:45] <paladox>	 !log repool mw1
[04:07:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:08:01] <paladox>	 !log depool mw2
[04:08:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:12:36] <paladox>	 !log repool mw2
[04:12:40] <paladox>	 !log depool mw3
[04:12:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:12:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:13:24] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:15:49] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3
[04:16:02] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.009 second response time
[04:16:07] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 309 bytes in 0.294 second response time
[04:17:47] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy
[04:18:04] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15190 bytes in 0.005 second response time
[04:18:05] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 0.364 second response time
[04:18:32] <paladox>	 !log repool mw3
[04:18:41] <paladox>	 !log depool lizardfs6
[04:18:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:18:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:21:51] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. lizardfs6
[04:23:22] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. lizardfs6
[04:25:18] <paladox>	 !log repool lizardfs6
[04:25:19] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy
[04:25:22] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy
[04:25:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[04:25:47] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy
[05:46:44] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfIM
[05:46:46] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03cc5b94c - Update php.pp
[06:27:24] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3202 MB (13% inode=94%);
[06:50:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfLz
[06:50:11] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03eeaf025 - BOT: Updating services config for wikis
[07:00:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfLr
[07:00:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 031347ad9 - BOT: Updating services config for wikis
[09:27:50] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfmL
[09:27:52] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 03ae2edfe - add 3 file types to list requested in T5006, approved (non-executable)
[09:51:11] <magnulu>	 good morning all
[09:51:55] <magnulu>	 will changes made on Special:ManageWiki/Extensions be persistent after submitting changes, or will these changes need to be made my someone manually?
[09:52:47] <magnulu>	 unsure if I messed up or sucessfully submitted my changes... 
[09:58:17] <Reception123>	 hi magnulu 
[09:59:39] <Reception123>	 magnulu: Yes, once you submit changes to MW/Extensions they will take effect in about 5-10 minutes.
[09:59:42] <Reception123>	 So please be patient :)
[10:18:55] <Reception123>	 !log sudo -u www-data php fixStuckGlobalRename.php --wiki=aawikiwiki PaperClip "FireBarrier101" --logwiki=metawiki
[10:19:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[10:29:39] <Reception123>	 !log run fixStuckGlobalRename.php on all wikis for username above
[10:29:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[11:12:17] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki GasMask0217 "KP619" --logwiki=metawiki
[11:12:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:00:04] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[12:01:58] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[12:08:52] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 52%
[12:10:51] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 27%
[13:14:41] <RhinosF1>	 .wmca Superzerocool
[13:14:41] <ZppixBot>	 https://meta.wikimedia.org/wiki/Special:CentralAuth/Superzerocool
[13:33:29] <RhinosF1>	 SPF|Cloud: Can I get a quick CU pls?
[13:33:44] <MH-Discord>	 <RhinosF1> @Stewards
[13:54:54] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins]
[14:04:46] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[14:06:19] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_landing]
[14:06:32] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[glusterfs.pem]
[14:06:51] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[glusterfs.pem]
[14:06:54] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[reviwiki.info_private]
[14:07:03] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[vitriol.spoon.army_private]
[14:11:54] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[14:13:05] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:13:30] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:13:48] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[14:22:21] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[14:52:05] <Reception123>	 !log reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki --delete Reception123
[14:52:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:58:58] <Reception123>	 !log DELETED and DROPPED dbs https://phabricator.miraheze.org/P247
[14:58:58] <ZppixBot>	 [ ✎ P247 deleted and dropped 11/01/2020 ] - phabricator.miraheze.org
[14:59:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:02:01] <icinga-miraheze>	 PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Mon 27 Jan 2020 02:58:36 PM GMT +0000).
[15:02:14] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfsN
[15:02:16] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 035ad5586 - Bot: Update SSL cert for vitriol.spoon.army
[15:12:02] <icinga-miraheze>	 RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Fri 10 Apr 2020 02:02:08 PM GMT +0000.
[15:24:01] <paladox>	 Reception123 they take 1minute to take effect :)
[15:24:24] <Reception123>	 paladox: oh
[15:29:29] <RhinosF1>	 hi paladox 
[15:29:36] <paladox>	 hi
[15:42:08] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[15:47:35] <Not-5a0a>	 [02MirahezeMagic] 07RhinosF1 opened pull request 03#95: Miraheze-ifying ca admin merge message - 13https://git.io/JvfZB
[15:50:03] <Not-5a0a>	 [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#95: Miraheze-ifying ca admin merge message - 13https://git.io/JvfZB
[15:51:07] <Not-5a0a>	 [02MirahezeMagic] 07RhinosF1 edited pull request 03#95: Miraheze-ifying ca admin merge message - 13https://git.io/JvfZB
[15:51:17] <RhinosF1>	 Reception123, paladox: ^
[16:10:37] <hispano76>	 hi
[16:10:43] <RhinosF1>	 hi hispano76 
[16:24:15] <magnulu>	 Reception123: thanks for the answer earlier :-)
[16:24:34] <Reception123>	 magnulu: you're welcome
[16:26:32] <RhinosF1>	 Reception123: there's that PR and a stuck rename if you've got access
[16:26:46] <magnulu>	 Reception123: I probably missed something, as they were not applied.. are these settings applied automatically or manually?
[16:27:01] <Reception123>	 magnulu: it is automatic once you click submit 
[16:27:10] <Reception123>	 magnulu: could you give us the URL so we can take a look?
[16:27:36] <magnulu>	 https://tolkewiki.miraheze.org - but I submitted them again 1 minute ago
[16:27:38] <ZppixBot>	 [ Tolkewiki ] - tolkewiki.miraheze.org
[16:27:40] <Not-5a0a>	 [02MirahezeMagic] 07Reception123 closed pull request 03#95: Miraheze-ifying ca admin merge message - 13https://git.io/JvfZB
[16:27:42] <Not-5a0a>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvfZx
[16:27:43] <Not-5a0a>	 [02miraheze/MirahezeMagic] 07RhinosF1 03b0e34eb - Miraheze-ifying ca admin merge message (#95) * Miraheze-ifying ca admin merge message * add i18n for additional ca override
[16:28:42] <magnulu>	 Reception123: and now all is good in the hood
[16:28:52] <Reception123>	 magnulu: great :) 
[16:28:59] <Reception123>	 magnulu: if you have any other issues or questions feel free to ask us here
[16:29:12] <RhinosF1>	 Reception123: don't forget to update in mediawiki repo and do i18n
[16:29:41] <Reception123>	 RhinosF1: yup 
[16:29:53] <magnulu>	 Reception123: sure, thanks :-)
[16:29:53] <Reception123>	 haven't updated something in the mediawiki repo for ages :)
[16:30:39] <RhinosF1>	 Reception123: see my guide on meta
[16:30:59] <Reception123>	 RhinosF1: I still remember how to but thanks for the offer 
[16:31:10] <Reception123>	 I've done it so many times in the past it's not something I'd forget ;)
[16:31:26] <Reception123>	 need to reclone mediawiki though so it will take a bit 
[16:31:27] <RhinosF1>	 :)
[16:31:31] <RhinosF1>	 true
[16:32:00] <RhinosF1>	 we need to automate extension updates
[16:33:09] <Reception123>	 could do
[16:34:06] <RhinosF1>	 maint script running can be done but not sure about the submodule thing
[16:34:15] <RhinosF1>	 paladox: thoughts?
[16:34:23] <Reception123>	 that would be complicated but the current GH bot could do it
[16:34:24] <paladox>	 That'll be dangerous to do.
[16:34:26] <RhinosF1>	 it'd have to be sense checked each time though
[16:34:32] <Reception123>	 paladox: why dangerous? it would just be an update
[16:34:48] <paladox>	 someone could merge some virus
[16:34:50] <RhinosF1>	 paladox: one that's activated by a sysadmin and watched
[16:34:55] <paladox>	 and we would have it deployed automatically
[16:35:00] <RhinosF1>	 but automates process
[16:35:08] <RhinosF1>	 so reduces clicks
[16:35:31] <Reception123>	 good point, it could increase risks 
[16:35:35] <Reception123>	 plus sometimes we don't want to deploy immediately
[16:35:51] <paladox>	 "automate extension updates" how, what are your thoughts exactly?
[16:36:42] <RhinosF1>	 paladox: so once a pr is done, someone logs onto one server and runs a script that shows extensions to be updated, agrees to do it and then does all the maint scripts
[16:36:55] <RhinosF1>	 so it's one command and not however many it is now
[16:36:58] <paladox>	 oh
[16:37:27] <paladox>	 that's not automating
[16:38:03] <RhinosF1>	 Semi-automating
[16:38:14] <paladox>	 it's not even semi automating
[16:38:14] <RhinosF1>	 still with the sysadmin oversight just less clicks
[16:38:24] <paladox>	 we just need to merge a pr and bingo it'll be deployed
[16:40:02] <RhinosF1>	 paladox: but that wouldn't cut out the submodule step, I'm saying a script that lists submodules to be updated and says would you like to update x from y to z and then you say yes and once you've done all needed ones it pushes to github, runs puppet and updates i18n cache
[16:41:31] <paladox>	 ok
[16:50:33] <Not-5a0a>	 [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jvfn3
[16:50:34] <Not-5a0a>	 [02miraheze/mediawiki] 07Reception123 03b5d6e31 - Update MirahezeMagic
[16:57:42] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki  on mw* and lizardfs6
[16:57:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:58:23] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 25004 MB (6% inode=95%);
[16:58:31] <paladox>	 !log root@bacula1:/bacula/backup# rm DB*
[16:58:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:59:53] <icinga-miraheze>	 RECOVERY - bacula1 Disk Space on bacula1 is OK: DISK OK - free space: / 119231 MB (25% inode=99%);
[17:03:00] <RhinosF1>	 Reception123: cache isnt purged
[17:03:06] <RhinosF1>	 On i18n
[17:03:45] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:03:55] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb
[17:03:56] <Reception123>	 RhinosF1: don't know why since I ran it
[17:04:24] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb
[17:04:46] <paladox>	 !log apt-get upgrade - bacula1
[17:04:57] <RhinosF1>	 Reception123: see https://meta.miraheze.org/wiki/Special:CentralAuth?target=FireBarrier101 - click the little ? Next to a hat
[17:05:01] <ZppixBot>	 [ Global account information for FireBarrier101 - Miraheze Meta ] - meta.miraheze.org
[17:05:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:05:47] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18673 bytes in 2.069 second response time
[17:06:24] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[17:09:39] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:19:19] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 4396 files, 9.233MB, 2020-01-11 17:17:00 (2.3 minutes ago)
[17:20:25] <Reception123>	 RhinosF1: don't know why...
[17:23:13] <RhinosF1>	 Reception123: hmm
[17:23:17] <RhinosF1>	 paladox: any idea?
[17:23:35] <paladox>	 ?
[17:23:53] <Reception123>	 paladox: about why after reloading LC the i18n doesn't work
[17:24:56] <paladox>	 oh
[17:29:03] <paladox>	 sadly nope :(
[17:29:30] <RhinosF1>	 Reception123: did it fully run? what was output?
[17:29:33] <paladox>	 well
[17:29:35] <paladox>	 actually
[17:29:35] <paladox>	 https://github.com/miraheze/MirahezeMagic/compare/b9c977175c4ba67aee0fc40866039cd121669cb7...b0e34eb0fab760f4dcaf8d327e5d57d35c9ef1ae#diff-a68cb4325f548c7238aeb7111385a5c1R79
[17:29:37] <ZppixBot>	 [ Comparing b9c977175c4ba67aee0fc40866039cd121669cb7...b0e34eb0fab760f4dcaf8d327e5d57d35c9ef1ae · miraheze/MirahezeMagic · GitHub ] - github.com
[17:29:37] <paladox>	 prefixes
[17:29:49] <Reception123>	 RhinosF1: as normal
[17:30:00] <Not-5a0a>	 [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvfce
[17:30:01] <Not-5a0a>	 [02miraheze/MirahezeMagic] 07paladox 03fc646e5 - Fix i18n
[17:30:22] <RhinosF1>	 paladox: ah
[17:30:26] <RhinosF1>	 Reception123: ok
[17:30:40] <Not-5a0a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jvfcv
[17:30:42] <Not-5a0a>	 [02miraheze/mediawiki] 07paladox 0336f91c7 - Update MirahezeMagic
[17:33:48] <paladox>	 RhinosF1 Reception123 works now
[17:34:04] <Reception123>	 paladox: ok, thanks
[17:34:44] <hispano76>	 https://github.com/miraheze/mw-config/pull/2853 Is there a problem you haven't seen? I'd like to implement it in the pages I'm making
[17:34:45] <ZppixBot>	 [ enable wgExtraInterlanguageLinkPrefixes on hispanowiki and Ucroniaswiki per request by Hispano76 · Pull Request #2853 · miraheze/mw-config · GitHub ] - github.com
[17:35:52] <RhinosF1>	 Thx paladox
[17:39:11] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb
[17:45:05] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[18:12:20] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfCU
[18:12:21] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03c5175c6 - Redis: Add loggroup
[18:15:11] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfCI
[18:15:13] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03b143c89 - Redis: Doin't log dupes
[18:35:43] <RhinosF1>	 hi JohnLewis 
[18:38:15] <JohnLewis>	 hi
[18:38:47] <RhinosF1>	 JohnLewis: pls see your email
[18:41:01] <icinga-miraheze>	 PROBLEM - wiki.serwerwanilia.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.serwerwanilia.pl' expires in 15 day(s) (Mon 27 Jan 2020 06:38:44 PM GMT +0000).
[18:41:14] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfC1
[18:41:16] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 039cdbdb9 - Bot: Update SSL cert for wiki.serwerwanilia.pl
[18:48:54] <hispano76>	 m
[18:49:18] <RhinosF1>	 hispano76: ?
[18:52:33] <icinga-miraheze>	 RECOVERY - wiki.serwerwanilia.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.serwerwanilia.pl' will expire on Fri 10 Apr 2020 05:41:08 PM GMT +0000.
[18:57:03] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_gdnsd_datacenters],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run]
[18:59:03] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:02:55] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[19:02:56] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:30:31] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45%
[19:32:28] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 28%
[19:54:15] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 43%
[19:56:16] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 24%
[20:01:46] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 1007552 files, 44.08GB, 2020-01-11 20:01:00 (45.0 seconds ago)
[20:01:46] <RhinosF1>	 paladox: you able to create a patch for https://phabricator.wikimedia.org/T242520#5794933
[20:01:47] <ZppixBot>	 [ ⚓ T242520 Allow Cloud mailing list to be indexed ] - phabricator.wikimedia.org
[20:01:50] <RhinosF1>	 ??
[20:02:15] <paladox>	 no? I doin't deal with the mailing list.
[20:02:26] <paladox>	 so i wouldn't know how to do that :)
[20:02:40] <paladox>	 oh
[20:02:48] <paladox>	 reading more you give the instructions
[20:03:08] <paladox>	 RhinosF1 gerrit has an inline editor, use it :)
[20:03:15] <RhinosF1>	 paladox: it does?
[20:03:20] <paladox>	 yes
[20:07:59] <RhinosF1>	 paladox: i created a change, now what?
[20:08:09] <paladox>	 you click the edit button
[20:12:36] <RhinosF1>	 paladox: I found it! How do you turn off WIP?
[20:12:46] <RhinosF1>	 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563684/
[20:12:46] <ZppixBot>	 [ Gerrit Code Review ] - gerrit.wikimedia.org
[20:13:30] <Reception123>	 RhinosF1: discovering the complicated ways of Gerrit?
[20:13:58] <RhinosF1>	 Reception123: yes
[20:14:05] <paladox>	 RhinosF1 press the start review thing
[20:15:08] <RhinosF1>	 paladox: I did gerrit!
[20:15:18] <paladox>	 congrats
[20:19:22] <RhinosF1>	 paladox: can you find a phab admin to block https://phabricator.wikimedia.org/p/Cutmuetia1998/
[20:19:23] <ZppixBot>	 [ ♟ Cutmuetia1998 ] - phabricator.wikimedia.org
[20:43:00] <Not-5a0a>	 [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvf8L
[20:43:01] <Not-5a0a>	 [02miraheze/mw-config] 07Pix1234 03acac341 - update wgCreateWikiNotificationEmail to use tech@
[21:11:00] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Databases db5 on bacula1 is OK: OK: Full, 2212 files, 61.69GB, 2020-01-11 21:09:00 (2.0 minutes ago)
[21:17:09] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 81154 files, 2.852GB, 2020-01-11 21:17:00 (8.0 seconds ago)
[22:34:38] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 43%
[22:36:36] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 29%
[22:48:06] <paladox>	 !log resizing misc3 - https://clientarea.ramnode.com/index.php?rp=/announcements/431/More-SSD-for-Standard-KVM.html
[22:48:07] <ZppixBot>	 [ News - RamNode ] - clientarea.ramnode.com
[22:48:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:52:27] <paladox>	 !log upgrade nodejs on misc3
[22:52:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:02:53] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[23:08:46] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:11:43] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[23:13:38] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[23:20:17] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:20:19] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:20:35] <icinga-miraheze>	 PROBLEM - misc2 Redis Process on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:21:24] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:21:46] <icinga-miraheze>	 PROBLEM - misc2 Current Load on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:21:48] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[23:21:50] <icinga-miraheze>	 PROBLEM - misc2 Disk Space on misc2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:22:03] <icinga-miraheze>	 PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:22:03] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:22:08] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host
[23:22:08] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host
[23:22:10] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[23:22:13] <icinga-miraheze>	 PROBLEM - misc2 SSH on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:22:29] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:22:37] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:23:07] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:23:18] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 1.163 second response time
[23:23:20] <icinga-miraheze>	 PROBLEM - Host misc2 is DOWN: PING CRITICAL - Packet loss = 100%
[23:23:21] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:23:31] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[23:23:32] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3
[23:23:56] <icinga-miraheze>	 RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18666 bytes in 0.270 second response time
[23:25:42] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18660 bytes in 1.014 second response time
[23:25:43] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3
[23:28:11] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.991 second response time
[23:29:00] <icinga-miraheze>	 RECOVERY - Host misc2 is UP: PING OK - Packet loss = 0%, RTA = 0.06 ms
[23:29:16] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures
[23:29:24] <icinga-miraheze>	 RECOVERY - misc2 Redis Process on misc2 is OK: PROCS OK: 1 process with args 'redis-server'
[23:29:50] <icinga-miraheze>	 RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 0.38, 0.35, 0.16
[23:29:59] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.695 second response time
[23:30:01] <icinga-miraheze>	 RECOVERY - misc2 Disk Space on misc2 is OK: DISK OK - free space: / 50369 MB (86% inode=98%);
[23:30:01] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.496 second response time
[23:30:03] <icinga-miraheze>	 RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.304 second response time
[23:30:08] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15196 bytes in 4.682 second response time
[23:30:14] <icinga-miraheze>	 RECOVERY - misc2 SSH on misc2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0)
[23:30:52] <paladox>	 !log switched TUN off on misc2
[23:30:58] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy
[23:31:10] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy
[23:32:04] <hispano76>	 :)
[23:33:18] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15190 bytes in 0.860 second response time
[23:33:45] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18659 bytes in 1.137 second response time
[23:34:17] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy
[23:34:34] <paladox>	 !log apt-get upgrade on misc2
[23:34:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:35:13] <paladox>	 !log [23:30:52]  <+paladox>	!log switched TUN off on misc2
[23:35:20] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[23:35:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:36:17] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:36:38] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvf08
[23:36:40] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0369c19ba - Redis: Increase timeout to 3