[00:10:22] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPT2
[00:10:24] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 030563452 - BOT: Updating services config for wikis
[00:12:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.31, 22.88, 18.88
[00:14:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.62, 21.68, 18.89
[00:16:25] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.74, 23.11, 19.73
[00:20:18] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.75, 21.58, 20.17
[00:22:15] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.77, 19.58, 19.57
[00:26:10] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.18, 21.03, 20.13
[00:28:06] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.34, 17.93, 19.10
[00:34:36] <Not-3033>	 [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JJPk5
[00:34:37] <Not-3033>	 [02miraheze/ssl] 07Southparkfan 03b5429c9 - Update and rename tla.awiki.org.crt to tl.awiki.org.crt T6005
[00:35:19] <Not-3033>	 [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPkF
[00:35:20] <Not-3033>	 [02miraheze/ssl] 07Southparkfan 030eb9b96 - tla.awiki.org -> tl.awiki.org Per T6005
[00:40:23] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPIe
[00:40:25] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 03fb43e8b - BOT: Updating services config for wikis
[00:47:36] <Not-3033>	 [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JJPIO
[00:47:38] <Not-3033>	 [02miraheze/ssl] 07Southparkfan 031ff04b6 - Add certificate for wiki.kali-team.cn
[00:48:47] <Not-3033>	 [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPIG
[00:48:49] <Not-3033>	 [02miraheze/ssl] 07Southparkfan 031ed4695 - Add wiki.kali-team.cn to certs.yaml
[00:52:53] <icinga-miraheze>	 PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:53:24] <Not-3033>	 [02miraheze/ssl] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPIW
[00:53:26] <Not-3033>	 [02miraheze/ssl] 07Southparkfan 03e479b82 - Fix enormous mistake in certs.yaml
[00:53:27] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:53:28] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:53:30] <icinga-miraheze>	 PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:53:39] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:53:46] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:54:04] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:54:09] <icinga-miraheze>	 PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:54:16] <SPF|Cloud>	 shut up icinga
[00:54:23] <icinga-miraheze>	 PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:54:38] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:54:51] <icinga-miraheze>	 RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[00:56:07] <icinga-miraheze>	 RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:56:23] <icinga-miraheze>	 RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:57:37] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[00:57:59] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[00:59:19] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:59:30] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:59:32] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:00:25] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPIH
[01:00:26] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 03efdb3ed - BOT: Updating services config for wikis
[01:02:24] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[01:03:18] <icinga-miraheze>	 RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[01:08:05] <MirahezeBot>	 Hello guest64! If you have any questions, feel free to ask and someone should answer soon.
[02:27:42] <hispano76>	 spam on -cvt
[02:34:01] <Cocopuff2018>	 Hello
[02:35:05] <hispano76>	 hi
[02:36:09] <Cocopuff2018>	 Ur still on
[03:05:22] <icinga-miraheze>	 PROBLEM - holonet.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holonet.pw' expires in 15 day(s) (Sun 23 Aug 2020 03:02:30 GMT +0000).
[03:05:52] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPOa
[03:05:54] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03faab376 - Bot: Update SSL cert for holonet.pw
[03:08:33] <icinga-miraheze>	 PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Sun 23 Aug 2020 03:01:19 GMT +0000).
[03:12:15] <icinga-miraheze>	 RECOVERY - holonet.pw - LetsEncrypt on sslhost is OK: OK - Certificate 'holonet.pw' will expire on Thu 05 Nov 2020 02:05:46 GMT +0000.
[03:16:26] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPOx
[03:16:27] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03379cada - Bot: Update SSL cert for enc.for.uz
[03:19:03] <icinga-miraheze>	 PROBLEM - studentwiki.ddns.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studentwiki.ddns.net' expires in 15 day(s) (Sun 23 Aug 2020 03:10:50 GMT +0000).
[03:22:10] <icinga-miraheze>	 RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 05 Nov 2020 02:16:19 GMT +0000.
[03:25:06] <icinga-miraheze>	 PROBLEM - wiki.bonessdiving.club - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bonessdiving.club' expires in 15 day(s) (Sun 23 Aug 2020 03:23:05 GMT +0000).
[03:25:30] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJP3g
[03:25:32] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 030275589 - Bot: Update SSL cert for studentwiki.ddns.net
[03:30:10] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJP37
[03:30:11] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 038433eb2 - Bot: Update SSL cert for wiki.bonessdiving.club
[03:30:58] <icinga-miraheze>	 PROBLEM - wiki.campaign-labour.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.campaign-labour.org' expires in 15 day(s) (Sun 23 Aug 2020 03:28:49 GMT +0000).
[03:32:35] <icinga-miraheze>	 RECOVERY - studentwiki.ddns.net - LetsEncrypt on sslhost is OK: OK - Certificate 'studentwiki.ddns.net' will expire on Thu 05 Nov 2020 02:25:24 GMT +0000.
[03:35:25] <icinga-miraheze>	 PROBLEM - wiki.dobots.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dobots.nl' expires in 15 day(s) (Sun 23 Aug 2020 03:31:50 GMT +0000).
[03:37:47] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPsC
[03:37:49] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03d29a856 - Bot: Update SSL cert for wiki.campaign-labour.org
[03:41:33] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPs8
[03:41:35] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03bcf2e9b - Bot: Update SSL cert for wiki.dobots.nl
[03:42:09] <icinga-miraheze>	 PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ngscott.net' expires in 15 day(s) (Sun 23 Aug 2020 03:38:53 GMT +0000).
[03:44:29] <icinga-miraheze>	 RECOVERY - wiki.campaign-labour.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.campaign-labour.org' will expire on Thu 05 Nov 2020 02:37:41 GMT +0000.
[03:45:02] <icinga-miraheze>	 PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Sun 23 Aug 2020 03:41:04 GMT +0000).
[03:45:43] <icinga-miraheze>	 PROBLEM - wiki.pupilliam.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pupilliam.com' expires in 15 day(s) (Sun 23 Aug 2020 03:42:08 GMT +0000).
[03:45:49] <icinga-miraheze>	 RECOVERY - wiki.bonessdiving.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bonessdiving.club' will expire on Thu 05 Nov 2020 02:30:02 GMT +0000.
[03:50:06] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPsS
[03:50:07] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 030514eb2 - Bot: Update SSL cert for wiki.ngscott.net
[03:50:52] <icinga-miraheze>	 PROBLEM - wiki.rmbrk.sk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rmbrk.sk' expires in 15 day(s) (Sun 23 Aug 2020 03:46:18 GMT +0000).
[03:55:25] <icinga-miraheze>	 RECOVERY - wiki.dobots.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dobots.nl' will expire on Thu 05 Nov 2020 02:41:27 GMT +0000.
[03:55:44] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPGn
[03:55:46] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03cecc374 - Bot: Update SSL cert for wiki.whentheycry.org
[03:59:30] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPGP
[03:59:32] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 036d662f0 - Bot: Update SSL cert for wiki.pupilliam.com
[04:02:26] <icinga-miraheze>	 RECOVERY - wiki.ngscott.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ngscott.net' will expire on Thu 05 Nov 2020 02:49:59 GMT +0000.
[04:05:38] <icinga-miraheze>	 RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Thu 05 Nov 2020 02:55:38 GMT +0000.
[04:07:01] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPZv
[04:07:02] <Not-3033>	 [02miraheze/ssl] 07MirahezeSSLBot 03aef2508 - Bot: Update SSL cert for wiki.rmbrk.sk
[04:13:00] <icinga-miraheze>	 RECOVERY - wiki.pupilliam.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pupilliam.com' will expire on Thu 05 Nov 2020 02:59:24 GMT +0000.
[04:17:36] <icinga-miraheze>	 RECOVERY - wiki.rmbrk.sk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rmbrk.sk' will expire on Thu 05 Nov 2020 03:06:54 GMT +0000.
[06:06:50] <icinga-miraheze>	 RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 20 packages available for upgrade (0 critical updates).
[06:06:51] <icinga-miraheze>	 RECOVERY - mail1 APT on mail1 is OK: APT OK: 38 packages available for upgrade (0 critical updates).
[06:08:58] <icinga-miraheze>	 RECOVERY - db7 APT on db7 is OK: APT OK: 19 packages available for upgrade (0 critical updates).
[06:09:28] <icinga-miraheze>	 RECOVERY - test2 APT on test2 is OK: APT OK: 44 packages available for upgrade (0 critical updates).
[06:10:05] <icinga-miraheze>	 RECOVERY - puppet2 APT on puppet2 is OK: APT OK: 2 packages available for upgrade (0 critical updates).
[06:16:03] <icinga-miraheze>	 RECOVERY - gluster2 APT on gluster2 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[06:20:54] <icinga-miraheze>	 RECOVERY - cp9 APT on cp9 is OK: APT OK: 35 packages available for upgrade (0 critical updates).
[06:21:39] <icinga-miraheze>	 RECOVERY - cp3 APT on cp3 is OK: APT OK: 36 packages available for upgrade (0 critical updates).
[06:26:32] <icinga-miraheze>	 RECOVERY - mw7 APT on mw7 is OK: APT OK: 42 packages available for upgrade (0 critical updates).
[06:31:10] <icinga-miraheze>	 RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 20 packages available for upgrade (0 critical updates).
[06:34:50] <icinga-miraheze>	 RECOVERY - phab1 APT on phab1 is OK: APT OK: 36 packages available for upgrade (0 critical updates).
[06:40:11] <icinga-miraheze>	 RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[06:41:29] <icinga-miraheze>	 RECOVERY - ns2 APT on ns2 is OK: APT OK: 20 packages available for upgrade (0 critical updates).
[06:46:52] <icinga-miraheze>	 RECOVERY - ldap1 APT on ldap1 is OK: APT OK: 20 packages available for upgrade (0 critical updates).
[06:52:18] <icinga-miraheze>	 RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 42 packages available for upgrade (0 critical updates).
[07:00:10] <icinga-miraheze>	 RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 20 packages available for upgrade (0 critical updates).
[07:00:35] <MirahezeBot>	 Hello id_mirounga_zh-c! If you have any questions, feel free to ask and someone should answer soon.
[07:09:49] <id_mirounga_zh-c>	 Does miraheze have limits about sending password reset email? a certain number of emails per hour?
[07:16:23] <RhinosF1>	 id_mirounga_zh-c: it will be limited, yes
[07:18:01] <RhinosF1>	 id_mirounga_zh-c: are you having an issue?
[07:27:44] <id_mirounga_zh-c>	 i just request password reset many times, so how long will i have to wait to receive new mail? an hour later? tomorrow?
[07:29:28] <RhinosF1>	 id_mirounga_zh-c: it should come through fairly fast but you should only request one and it's probably a rolling or daily limit. Is the email not coming through? Is it failing? Have you checked spam?
[07:33:57] <id_mirounga_zh-c>	 i checked just now. i use bitwarden and sign up recently, i found something wrong about auto-fill, it seems system did not confirm password change or some other errors happened on my pc
[07:35:09] <RhinosF1>	 id_mirounga_zh-c: it would be a good idea normally to paste your password somewhere private and ensure auto fill & password managers update
[07:37:21] <RhinosF1>	 If you've reset your password then immediately forgot, I'm not sure what we can do.
[07:42:43] <id_mirounga_zh-c>	 in fact i use ctrl-v paste password all the step, after i use code in email and changed password, then i log out, but system do not let me in with same password. i use MS Edge now, i think it is the problem, maybe
[07:44:07] <RhinosF1>	 id_mirounga_zh-c: try another browser but have you ensured what's on your clipboard is correct
[07:44:16] <RhinosF1>	 It should be the new password you set
[07:45:21] <Not-3033>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPEM
[07:45:23] <Not-3033>	 [02miraheze/CreateWiki] 07translatewiki 0359ada4c - Localisation updates from https://translatewiki.net.
[07:45:24] <MirahezeBot>	 [ Main page - translatewiki.net ] - translatewiki.net
[07:45:24] <Not-3033>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJPED
[07:45:26] <Not-3033>	 [02miraheze/MirahezeMagic] 07translatewiki 036d55131 - Localisation updates from https://translatewiki.net.
[07:45:27] <MirahezeBot>	 [ Main page - translatewiki.net ] - translatewiki.net
[07:49:07] <id_mirounga_zh-c>	 yes, so i am waiting the mail then change to firefox. to check the issue, i just want to know how long i have to wait, is it included in your docs?
[07:51:11] <RhinosF1>	 id_mirounga_zh-c: you have to wait 24 hours between sending reset emails
[07:59:59] <id_mirounga_zh-c>	 thanks for your reply<3  i can do other works now ,leave it to tomorrow
[08:15:20] <icinga-miraheze>	 PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 94%
[08:15:49] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.14, 21.16, 16.16
[08:16:15] <icinga-miraheze>	 PROBLEM - cp6 Current Load on cp6 is CRITICAL: CRITICAL - load average: 16.22, 14.46, 6.33
[08:16:50] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 16.73, 24.63, 12.21
[08:17:17] <icinga-miraheze>	 RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 31%
[08:19:38] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 19.16, 21.68, 17.54
[08:20:08] <icinga-miraheze>	 PROBLEM - cp6 Current Load on cp6 is WARNING: WARNING - load average: 0.70, 7.04, 5.09
[08:20:27] <RhinosF1>	 Reception|away: no internet?
[08:21:23] <RhinosF1>	 Traffic surge
[08:21:33] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.82, 19.06, 17.05
[08:22:04] <icinga-miraheze>	 RECOVERY - cp6 Current Load on cp6 is OK: OK - load average: 0.62, 4.97, 4.56
[08:28:05] <Reception|away>	 RhinosF1: not at home anyway but yeah still nothing :(
[08:29:08] <RhinosF1>	 Reception|away: i can't access cp* anyway even if I had much time anyway
[08:29:40] <Reception|away>	 Yeah, though you should see stuff on mw*
[08:32:50] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 0.73, 4.44, 7.44
[08:34:49] <icinga-miraheze>	 RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 0.42, 3.15, 6.62
[08:44:13] <RhinosF1>	 Reception|away: possibly
[13:27:32] <Cocopuff2018>	 hello?
[13:28:18] <hispano76>	 hi
[14:35:54] * hispano76 greetings
[14:49:07] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb
[14:49:53] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb
[14:50:04] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:50:09] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.138 second response time
[14:50:11] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:50:14] <icinga-miraheze>	 PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 10.45, 7.61, 4.60
[14:51:08] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:51:14] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.82, 20.08, 17.53
[14:51:17] <icinga-miraheze>	 PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 12.16, 8.54, 4.97
[14:51:51] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[14:51:58] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.016 second response time
[14:52:06] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.012 second response time
[14:52:08] <icinga-miraheze>	 RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.037 second response time
[14:53:03] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[14:53:03] <icinga-miraheze>	 RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15653 bytes in 0.004 second response time
[14:53:17] <icinga-miraheze>	 RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.11, 6.77, 4.76
[14:54:07] <icinga-miraheze>	 RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 3.26, 6.31, 4.86
[14:55:05] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.15, 21.43, 18.63
[14:57:00] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.37, 18.89, 18.07
[16:25:12] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJXkn
[16:25:13] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 038e939a5 - BOT: Updating services config for wikis
[16:42:08] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.20, 20.89, 18.29
[16:45:59] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.20, 19.20, 18.24
[17:26:43] <Jakeukalane>	 hello
[17:27:01] <Jakeukalane>	 our page takes a lot of time to load but the other wiki don't
[17:27:07] <Jakeukalane>	 we think is a complement
[17:27:11] <Jakeukalane>	 maybe the disqus tag
[17:27:17] <Jakeukalane>	 how can I ask to remove it?
[17:28:08] <Jakeukalane>	 our page is https://tuscriaturas.miraheze.org/wiki/El_Bestiario_del_Hypogripho_Dorado
[17:28:32] <MirahezeBot>	 [ Bestiario del Hypogripho ] - tuscriaturas.miraheze.org
[17:29:58] <Jakeukalane>	 the extension I think is the culprit is DisqusTag
[17:30:06] <Jakeukalane>	 but I don't know for sure
[17:31:23] <RhinosF1>	 Jakeukalane: I will log into my pc and check the cause
[17:31:34] <Jakeukalane>	 thank you
[17:34:56] <RhinosF1>	 Give me half an hour
[17:40:29] <RhinosF1>	 Jakeukalane: i see it
[17:47:39] <RhinosF1>	 Jakeukalane: something doesn't add up, I think there's latency somewhere.
[17:49:43] <Jakeukalane>	 maybe is one of the complements. One of the admins see a TLS shaking taking very long, and also a "connecting with disqus" that takes a lot. But I don't know if it is that because when I tried to do a profiling with Chrome it failed.
[17:49:48] <Jakeukalane>	 I will try to do another
[17:50:06] <RhinosF1>	 i think it's the connection to disqus
[17:50:13] <RhinosF1>	 SPF|Cloud: ^
[17:50:49] <Jakeukalane>	 then it would be solved if disqustag is removed right?
[17:51:17] <RhinosF1>	 Jakeukalane: yes
[17:52:34] <Jakeukalane>	 then I request the remove of that extension (we aren't using it either way)
[17:52:51] <RhinosF1>	 i can do that
[17:55:10] <RhinosF1>	 Jakeukalane: done
[17:58:30] <Jakeukalane>	 thank you
[17:58:39] <Jakeukalane>	 the changes don't get reflected inmedialty?
[18:03:43] <hispano76>	 it may take a while or minutes
[18:15:14] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJXYg
[18:15:16] <Not-3033>	 [02miraheze/services] 07MirahezeSSLBot 03aa01a9c - BOT: Updating services config for wikis
[18:16:57] <RhinosF1>	 what hispano76 said
[19:27:36] <SPF|Cloud>	 RhinosF1: I'm curious what makes you think disqus is the issue
[19:29:04] <RhinosF1>	 SPF|Cloud: because I looked in the network tab and it showed a few slow loading scripts from disqus
[19:29:24] <RhinosF1>	 But then I compared the size and they were nowhere near huge files
[19:29:32] <SPF|Cloud>	 that could be the symptom of a network issue between you and disqus
[19:30:05] <RhinosF1>	 SPF|Cloud: when the page loading slow affects multiple users at different locations
[19:30:07] <SPF|Cloud>	 but is that the reason a page takes over 20 seconds to parse at server level?
[19:30:20] <RhinosF1>	 Hmm
[19:30:38] <RhinosF1>	 What's happened their
[19:32:20] <SPF|Cloud>	 profiling the page tells me 99.16% of the time is spent in Message::toString
[19:32:42] <SPF|Cloud>	 now that's something to look at
[19:33:24] <RhinosF1>	 Oh?
[19:33:54] <RhinosF1>	 If we could narrow down further then that would be interesting
[19:34:08] <RhinosF1>	 Is it a specific message? A group of messages?
[19:34:13] <SPF|Cloud>	 have you seen my mail regarding jobrunners?
[19:34:35] <RhinosF1>	 SPF|Cloud: have you seen my reply?
[19:34:44] <SPF|Cloud>	 I guess not...
[19:34:47] * SPF|Cloud looks
[19:36:06] <SPF|Cloud>	 seen
[19:36:45] <SPF|Cloud>	 the redis jobqueue code still exists in master, and I cannot find a deprecation notice
[19:38:03] <SPF|Cloud>	 I am aware Wikimedia uses an EventBus based system, but there are more MediaWiki functions Wikimedia doesn't use
[19:38:18] <SPF|Cloud>	 but that doesn't have to mean that code is entirely unsupported
[19:41:56] <RhinosF1>	 SPF|Cloud: I was going to try and talk to CPT about the job queue system to try and get a better understanding of Kafka v redis and how a migration should work so I'm thinking it might be good to mention your email and see if we can get a full picture of timings. Sound good?
[19:42:54] <SPF|Cloud>	 you can, but merely to gain insight in the Kafka based solution for a future project
[19:44:00] <SPF|Cloud>	 this must not be a case of "spend 50 hours to migrate to a new system, knowing Wikimedia handled far, far more jobs with the Redis based queue"
[19:44:15] <RhinosF1>	 SPF|Cloud: ok
[19:45:34] <JohnLewis>	 Migration is relatively simple though - it's just changing the config no?
[19:49:10] <RhinosF1>	 The person who i think is charge of this at WMF is online, just speaking to them
[19:49:33] <RhinosF1>	 SPF|Cloud: 20:49:08 <Pchelolo> ok. so we've replaced redis-based queue with kafka-based queue specifically for this reason, we've had tons of jobs stuck and other problems
[19:51:33] <SPF|Cloud>	 hm, I must be missing something there - is kafka doing more than redis? to my knowledge redis is only used to keep track of jobs, with the jobrunner service being responsible for executing non-abadoned jobs
[19:52:51] <SPF|Cloud>	 I guess the question "is kafka doing more than redis?" is essential here for my understanding; if the issue was in redis, that sounds like there are bugs in redis, but I doubt Wikimedia would see that as a reason to migrate :)
[19:53:26] <RhinosF1>	 SPF|Cloud: it seems they knew about it then, trying to see about a support timeline and they've said it's designed a bit WMF internal as it needs so many parts but they want to make it have less moving parts
[19:53:43] <RhinosF1>	 And yeah just got back to say that CPT weren't really supporting it
[19:55:18] <Not-3033>	 [02landing] 07JohnFLewis closed pull request 03#38: Update to bootstrap 4.5.0 - 13https://git.io/Jf973
[19:55:58] <SPF|Cloud>	 aha, got that
[19:58:10] <SPF|Cloud>	 looking at the wikitech page, the Wikimedia setup seems overkill for us, albeit we don't need things like DC redundancy
[19:58:55] <SPF|Cloud>	 so I wonder if the setup process can be simplified by a huge margin
[19:59:36] <RhinosF1>	 SPF|Cloud: probably, see -staff
[20:10:49] <Cocopuff2018>	 Did that user amanda Catherin Retire from miraheze?
[20:12:59] <Cocopuff2018>	 @rhinosF1 from what i have read there is some unhappy users with the system admin
[20:13:22] <RhinosF1>	 She did
[20:13:40] <RhinosF1>	 And Reception|away has replied (since removed) on her talk page
[20:15:25] <Cocopuff2018>	 she did leave a message on her userpage aswell
[20:16:41] <Cocopuff2018>	 people need to understand  sys admin cannot make everyone happy, nor do the Policies need to be changed just for there Disagreement 
[20:20:48] <Cocopuff2018>	 if a wiki was deleted will it Removed it from the number of global edits aswell?
[20:21:04] <Cocopuff2018>	 remove the number of edits from that users global edits?
[20:24:33] <Cocopuff2018>	 @rhinosF1 @Dough 
[20:24:34] <Cocopuff2018>	 https://help.imdb.com/article/imdb/general-information/can-i-use-imdb-data-in-my-software/G5JTRESSHJBBHTGX?ref_=helpsrall#
[20:24:35] <MirahezeBot>	 [ IMDb | Help ] - help.imdb.com
[20:28:33] <RhinosF1>	 Cocopuff2018: after the database is dropped
[20:29:27] <Cocopuff2018>	 the wiki will not be an issue to imdb
[20:31:27] <RhinosF1>	 Ok
[20:43:14] <Jakeukalane>	 @rhinosF1 then it need more things to go faster? I already appreciate a bump up in speed
[20:44:52] <SPF|Cloud>	 Jakeukalane: is the wiki still slow for you now?
[20:46:13] <SPF|Cloud>	 I performed some tests on your wiki, and found the extension LinkTitles to be causing the absurd performance impact
[20:46:37] <SPF|Cloud>	 disabling the extension fixes the issue
[20:47:33] <Jakeukalane>	 no, it now loads a lot faster
[20:47:54] <Jakeukalane>	 linktitles is the one that links to the articles found in the text right?
[20:48:04] <SPF|Cloud>	 yes, seems so
[20:48:46] <SPF|Cloud>	 given the default settings, I'm not surprised it makes your wiki visibly slower
[20:53:32] <Jakeukalane>	 it is solved now, thank you very much. I was reticent to add linktitles, I was right
[20:53:52] <SPF|Cloud>	 in fact, one of our colleagues already found out in 2016 that "the more pages a wiki has, the worse the performance this extension will get." (https://phabricator.miraheze.org/T1160#21852)
[20:53:53] <MirahezeBot>	 [ ⚓ T1160 Installing extensions ] - phabricator.miraheze.org
[20:54:30] <SPF|Cloud>	 if you prefer to have LinkTitles enabled with assistance from us to optimize that extension, please open a Phabricator ticket
[20:54:44] <SPF|Cloud>	 and if you don't really need LinkTitles, I recommend to keep it disabled :)
[20:55:08] <icinga-miraheze>	 PROBLEM - ns1 Current Load on ns1 is CRITICAL: CRITICAL - load average: 0.95, 3.72, 2.29
[20:57:08] <icinga-miraheze>	 PROBLEM - ns1 Current Load on ns1 is WARNING: WARNING - load average: 0.02, 1.66, 1.77
[20:59:08] <icinga-miraheze>	 RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.00, 0.75, 1.37
[21:30:31] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.12, 19.60, 16.79
[21:32:26] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.48, 17.49, 16.39
[22:03:28] <icinga-miraheze>	 PROBLEM - wiki.lbcomms.co.za - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.lbcomms.co.za All nameservers failed to answer the query.
[22:04:00] <RhinosF1>	 SPF|Cloud: it worked ^
[22:04:37] <icinga-miraheze>	 PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:17:08] <icinga-miraheze>	 RECOVERY - wiki.lbcomms.co.za - reverse DNS on sslhost is OK: rDNS OK - wiki.lbcomms.co.za reverse DNS resolves to cp7.miraheze.org
[22:18:44] <icinga-miraheze>	 RECOVERY - wiki.lbcomms.co.za - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lbcomms.co.za' will expire on Mon 14 Sep 2020 09:10:41 GMT +0000.
[22:34:58] <Jakeukalane>	 is a extension that we can able and disable or is admin managed?
[22:50:48] <Cocopuff2018>	 if a wiki was deleted will it Removed it from the number of global edits from the users who edited the wiki aswell?
[22:55:04] <hispano76>	 yes
[22:55:25] <Cocopuff2018>	 thats what i thought
[22:55:28] <Cocopuff2018>	 (:
[22:55:38] <hispano76>	 :)
[22:56:39] <Cocopuff2018>	 someone put hi im testing a thing i put
[23:01:47] <Cocopuff2018>	 Anyone on?