[00:04:09] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb
[00:05:05] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis]
[00:05:11] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[00:05:52] <icinga-miraheze>	 PROBLEM - lizardfs6 Redis Process on lizardfs6 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server'
[00:06:18] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb
[00:07:16] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15182 bytes in 8.578 second response time
[00:10:01] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[00:10:14] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[00:13:49] <icinga-miraheze>	 RECOVERY - lizardfs6 Redis Process on lizardfs6 is OK: PROCS OK: 1 process with args 'redis-server'
[00:20:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfEI
[00:20:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 0324b9150 - BOT: Updating services config for wikis
[00:23:27] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:32:28] <hispano76>	 https://phabricator.miraheze.org/T4874#97074
[00:32:29] <ZppixBot>	 [ ⚓ T4874 Extension Review for Extension:SpriteSheet ] - phabricator.miraheze.org
[00:34:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfEc
[00:34:57] <Not-5a0a>	 [02miraheze/puppet] 07paladox 033c6c761 - redis: Tweak config
[00:35:46] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 44%
[00:39:38] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 19%
[01:00:02] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfEQ
[01:00:03] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 033010d4f - Redis: Cross out some configs Some of the configs will fallback to the value used by $wgMainCacheType. Based on some observations, doing this feel faster. We can revert if this makes things worse.
[01:20:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfuU
[01:20:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 032c9a7a2 - BOT: Updating services config for wikis
[06:26:07] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3203 MB (13% inode=94%);
[06:30:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfaR
[06:30:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03877b7f7 - BOT: Updating services config for wikis
[08:02:52] <Reception123>	 RhinosF1: around?
[08:04:46] <RhinosF1>	 Yes
[08:05:25] <Reception123>	 RhinosF1: https://twinkletest.miraheze.org/ I'm trying the simpler version of Twinkle again because the enwiki one is just too complex, but now I'm getting something even more annoying
[08:05:27] <ZppixBot>	 [ Twinkle Test Wiki ] - twinkletest.miraheze.org
[08:05:31] <Reception123>	 The gadgets tab doesn't even show up for me
[08:05:42] <Reception123>	 so I can't enable Twinkle
[08:05:49] <RhinosF1>	 Reception123: admin + crat me
[08:05:52] <Reception123>	 but I don't get why I've done everything
[08:06:07] <Reception123>	 RhinosF1: done
[08:06:50] <Reception123>	 !log ran rebuildall on twinkletestwiki 
[08:06:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[08:07:05] <RhinosF1>	 Reception123: try now
[08:07:34] <Reception123>	 RhinosF1: now that's embarrassing :P 
[08:07:40] <Reception123>	 I really thought Gadgets was a global extension
[08:08:14] <RhinosF1>	 Reception123: np
[08:08:44] <Reception123>	 now the question is why is there no Twinkle tab...
[08:08:47] <RhinosF1>	 Reception123: Nothing shows on main page for me. Anything in console?
[08:09:37] <Reception123>	 Exception in module-execute in module ext.gadget.Twinkle:
[08:11:10] <RhinosF1>	 Whats the exceptiom
[08:11:50] <Reception123>	 can't really tell there's like 400 lines of code there
[08:13:25] <RhinosF1>	 Reception123: have you imported morebits
[08:13:52] <Reception123>	 RhinosF1: nope, had no idea that was a requirement
[08:14:42] <Reception123>	 doing
[08:15:53] <Reception123>	 RhinosF1: perfect now it works thanks!
[08:15:59] <Reception123>	 RhinosF1: "Your account is too new to use Twinkle." now that's just great 
[08:17:30] <Reception123>	 At least I've found how to disable htat
[08:17:32] <Reception123>	 *that
[08:20:08] <RhinosF1>	 Reception123: hows it work now
[08:20:22] <Reception123>	 RhinosF1: Still trying to get rid of the account age restriction
[08:20:28] <Reception123>	 RhinosF1: do you know if // (comment out) works in JS?
[08:21:23] <RhinosF1>	 Reception123: it should
[08:21:43] <Reception123>	 guess I'll find out now
[08:22:24] <Reception123>	 JavaScript parse error: Parse error: 
[08:22:27] <Reception123>	 looks like it got mad
[08:27:54] <Reception123>	 RhinosF1: It works!!
[08:28:00] <Reception123>	 I think we've finally got it
[08:28:56] <Reception123>	 https://twinkletest.miraheze.org/wiki/User_talk:RhinosF1
[08:28:58] <ZppixBot>	 [ User talk:RhinosF1 - Twinkle Test Wiki ] - twinkletest.miraheze.org
[08:30:51] <RhinosF1>	 Reception123: great but mine wont purge and reconigse morebits. Can you run rebuildall and purgeList.php —all
[08:31:13] <RhinosF1>	 —all —purge even
[08:31:14] <Reception123>	 RhinosF1: did you follow the refresh/purge instructions on the js page?
[08:31:27] <Reception123>	 I had to do that in order for it to work for me so it could just be that
[08:32:10] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php purgeList.php --all --purge --wiki twinkletestwiki + rebuildall.php
[08:32:21] <RhinosF1>	 Reception123: ive just purged my cache but lets see
[08:32:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[08:32:55] <Reception123>	 RhinosF1: I'm also trying with my test account to see if speedy deletion requests work since from an admin account it just deletes
[08:34:18] <Reception123>	 RhinosF1: Okay, so now all we need to do is change it a bit so that it works for Meta (and hope we don't mess anything up) and it should be good to deploy 
[08:35:36] <RhinosF1>	 Ok
[08:35:48] <RhinosF1>	 I need to fix my angry cache
[08:37:18] <Reception123>	 ok, I've done what you asked as you can see so I don't know why
[08:37:40] <Reception123>	 I'm trying to see if we can remove some irrelevant tags for us and get it not to error
[08:43:32] <RhinosF1>	 Reception123: okay, im out for breakfast but’ll fiddle later
[08:43:42] <Reception123>	 RhinosF1: ok
[08:44:25] <RhinosF1>	 Shall we do a #miraheze-twinkle channel?
[08:44:52] <Reception123>	 RhinosF1: we can just PM if you want, it's not like anyone else is joining 
[09:45:14] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/local/bin/gen_fingerprints],File[/etc/vim/vimrc.local]
[09:51:23] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[13:10:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvf6u
[13:10:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 032b1999c - BOT: Updating services config for wikis
[14:58:16] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfPA
[14:58:17] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03f7c01d0 - Set wgParserCacheType to CACHE_DB Change-Id: Idaf5fbc5bb8d70419df98109462a23ef6efe2047
[15:12:08] <Not-5a0a>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfXs
[15:12:10] <Not-5a0a>	 [02miraheze/mw-config] 07Southparkfan 03d2cc9e5 - Enable message cache
[15:15:53] <paladox>	 !log temporarily depool lizardfs6
[15:16:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:19:20] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. lizardfs6
[15:23:08] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfXV
[15:23:09] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 031e07192 - Switch wgLanguageConverterCacheType back to DB
[15:23:14] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[15:23:30] <paladox>	 well heh, why did lfs6 go down, i depooled it
[15:23:34] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[15:23:53] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[15:23:55] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[15:25:15] <paladox>	 !log repooled lizardfs6
[15:25:31] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy
[15:25:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:25:52] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[15:27:05] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:32:15] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21969 MB (5% inode=95%);
[15:38:08] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 23967 MB (6% inode=95%);
[16:35:35] <icinga-miraheze>	 PROBLEM - glp.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'glp.ink' expires in 7 day(s) (Mon 20 Jan 2020 04:34:07 PM GMT +0000).
[18:53:12] * hispano76 greetings
[18:53:56] <RhinosF1>	 Hi hispano76
[18:56:06] <hispano76>	 hi RhinosF1 :)
[20:05:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfQY
[20:05:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 0368b54af - BOT: Updating services config for wikis
[21:21:47] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45%
[21:29:43] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 32%
[21:42:15] <RhinosF1>	 Vote: https://meta.miraheze.org/wiki/Community_noticeboard#Motion_to_delay_elections.2C_1_year
[21:42:16] <ZppixBot>	 [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org
[22:04:30] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21976 MB (5% inode=95%);
[22:15:56] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[contraao.com]
[22:23:51] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:23:58] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.55, 1.96, 1.16
[22:25:55] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.54, 1.26, 1.06
[22:40:34] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfFa
[22:40:35] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03be047c2 - nutcracker: Config tweek
[22:41:30] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfFw
[22:41:31] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03362f683 - Redis: Remove connectTimeout
[22:47:56] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvfFQ
[22:47:57] <Not-5a0a>	 [02miraheze/puppet] 07paladox 034c98afe - nutcracker: Switch to use sockets
[22:47:59] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS
[22:48:00] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1184: nutcracker: Switch to use sockets - 13https://git.io/JvfF7
[22:48:54] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvfFd
[22:48:55] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03865d5f0 - Update nutcracker.yml.erb
[22:48:57] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1184: nutcracker: Switch to use sockets - 13https://git.io/JvfF7
[22:49:10] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1184: nutcracker: Switch to use sockets - 13https://git.io/JvfF7
[22:49:11] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvfFN
[22:49:13] <Not-5a0a>	 [02miraheze/puppet] 07paladox 032d8af99 - nutcracker: Switch to use sockets (#1184) * nutcracker: Switch to use sockets * Update nutcracker.yml.erb
[22:50:21] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS
[22:50:23] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11
[22:53:49] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvfFj
[22:53:50] <Not-5a0a>	 [02miraheze/puppet] 07paladox 033e8fe7f - Revert "nutcracker: Switch to use sockets (#1184)" This reverts commit 2d8af997142924819a443a3286e8550cc2680ffa.
[22:56:00] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jvfbf
[22:56:01] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c253b8e - Revert "Revert "nutcracker: Switch to use sockets (#1184)"" This reverts commit 3e8fe7f5bde5d5a588991aff8bc1bce079264332.
[22:56:25] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvfbU
[22:56:26] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03aeee2eb - Redis: Switch to sockets
[23:02:06] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[23:25:38] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvfbw
[23:25:40] <Not-5a0a>	 [02miraheze/puppet] 07paladox 031506055 - Remove redis from all mediawiki hosts
[23:28:39] <icinga-miraheze>	 PROBLEM - mw1 Redis Process on mw1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server'
[23:29:07] <icinga-miraheze>	 PROBLEM - lizardfs6 Redis Process on lizardfs6 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server'
[23:29:12] <paladox>	 !log removed redis from all mediawiki servers
[23:29:27] <icinga-miraheze>	 PROBLEM - mw2 Redis Process on mw2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server'
[23:29:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:29:35] <icinga-miraheze>	 PROBLEM - mw3 Redis Process on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server'