[00:04:10] <icinga-miraheze>	 PROBLEM - private.revi.wiki - Comodo on sslhost is WARNING: WARNING - Certificate 'private.revi.wiki' expires in 30 day(s) (Thu 07 Nov 2019 11:59:59 PM GMT +0000).
[00:04:32] <revi>	 okay
[00:07:01] <Zppix>	 revi:  is it?
[00:07:15] <revi>	 it what?
[00:07:33] <Zppix>	 revi:  you said okay :P
[00:07:45] <revi>	 ofc I have to renew but it's okay per se :P
[00:10:37] <Zppix>	 xD
[02:12:37] <Examknow>	 Zppix: Hey
[02:14:18] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3
[02:16:03] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[02:16:15] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[02:17:09] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[02:18:36] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[02:18:44] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[02:36:41] <Zppix>	 Examknow:  whats up?
[02:37:20] <Examknow>	 Zppix: Just curious what do you use to host your bot?
[02:37:26] <Zppix>	 Examknow:  which one?
[02:37:35] <Examknow>	 ZppixBot
[02:37:53] <Examknow>	 I made mine with sopel but it only works whn my pc is online
[02:37:56] <Zppix>	 Examknow:  I use wikimedia toolforge but your bot is out of scope for toolforge so wikimedia wouldnt let you
[02:38:17] <Examknow>	 oh
[02:38:28] <Examknow>	 do you know of any alternitives?
[02:38:45] <Zppix>	 Examknow:  xshellz maybe? or heroku
[02:39:03] <Examknow>	 Xshellz does not work with sopel
[02:39:17] <Examknow>	 I am also looking for a preferably free service
[02:41:50] <Zppix>	 heroku is free
[02:41:53] <Zppix>	 i use it for MH-Discord 
[02:43:37] <Examknow>	 ah
[02:43:43] <Examknow>	 I will try that
[05:30:34] <ZppixBot>	 Hello Guest2107! If you have any questions, feel free to ask and someone should answer soon.
[06:06:42] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.2-apcu],Package[php7.2-redis]
[06:14:34] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[06:26:16] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3112 MB (12% inode=94%);
[08:45:08] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWqN
[08:45:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 0363aac6a - BOT: Updating services config for wikis
[12:58:31] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[13:00:22] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2
[13:02:58] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[13:04:54] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2
[13:04:56] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[13:06:19] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3
[13:06:29] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 6.753 second response time
[13:08:15] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[13:08:20] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[13:09:02] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[13:09:23] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[13:15:54] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2
[13:15:55] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[13:16:35] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[13:16:53] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[13:18:40] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[13:18:41] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[13:19:04] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[13:37:09] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.07, 6.65, 5.54
[13:40:14] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.84, 6.17, 5.54
[14:17:10] <ZppixBot>	 Hello RhinosF1_! If you have any questions, feel free to ask and someone should answer soon.
[14:19:51] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[14:22:36] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Diff, 60047 files, 58.10GB, 2019-09-15 01:59:00 (3.4 weeks ago)
[14:55:10] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:55:17] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[14:55:51] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[14:55:51] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1
[14:55:52] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[14:57:54] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.389 second response time
[14:57:58] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[14:58:22] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[14:58:22] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[14:58:27] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[14:58:27] <icinga-miraheze>	 PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.vinesh.eu.org' expires in 15 day(s) (Thu 24 Oct 2019 02:52:57 PM GMT +0000).
[14:58:42] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWng
[14:58:44] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot 0359bfa68 - Bot: Update SSL cert for wiki.vinesh.eu.org
[15:04:28] <icinga-miraheze>	 RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Mon 06 Jan 2020 01:58:36 PM GMT +0000.
[15:10:01] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1
[15:11:27] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[15:11:31] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[15:16:23] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2
[15:17:04] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[15:17:54] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:17:58] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[15:19:05] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[15:41:23] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 6.57, 5.32
[15:44:05] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.16, 6.12, 5.36
[15:54:17] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb
[15:54:21] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb
[15:57:01] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3
[15:59:47] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[15:59:51] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:59:57] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[16:23:46] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.38, 7.03, 6.21
[16:26:29] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.58, 6.13, 6.00
[16:43:29] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:48:29] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3
[16:48:33] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3
[16:48:40] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[16:49:01] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:50:11] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.008 second response time
[16:50:53] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:51:43] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[16:51:50] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[16:52:03] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[16:52:20] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[16:53:33] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:24:21] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 6.83, 6.08, 5.03
[17:26:49] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 5.42, 6.03, 5.17
[17:27:06] <ZppixBot>	 Hello rivero! If you have any questions, feel free to ask and someone should answer soon.
[17:36:01] <rivero>	 Hi. I'm not sure if this is the right place to ask about this, but I have (or had rather) a wiki that because of work/school I stopped attending to for a few months. I'm finally back in the place where I can edit and work on it, but it seems to have been deleted. Is there a way to recover it? Cheers
[17:40:43] <Voidwalker>	 rivero, what was the wiki? and was it public?
[17:41:18] <rivero>	 agoldenbraid.miraheze.org and it was public 
[17:42:06] <RhinosF1>	 rivero: when did you last use it?
[17:42:44] <rivero>	 Sometime in May/April of this year 
[17:43:47] <RhinosF1>	 rivero: can you create a task on Phabricator.miraheze.org and assign it to Reception123 asking for the backup to be restored if it exists
[17:44:03] <RhinosF1>	 And request the wiki again please from the same account
[17:46:40] <rivero>	 sure, will do
[18:05:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWl0
[18:05:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03388c741 - BOT: Updating services config for wikis
[18:15:09] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.26, 6.83, 5.86
[18:17:47] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.69, 6.74, 5.97
[18:23:37] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.09, 7.01, 6.36
[18:25:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWlr
[18:25:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03afa1bff - BOT: Updating services config for wikis
[18:28:44] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.25, 6.69, 6.40
[18:34:34] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.11, 7.52, 6.91
[18:37:48] <icinga-miraheze>	 PROBLEM - wiki.pyxxl.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pyxxl.tk' expires in 15 day(s) (Thu 24 Oct 2019 06:31:50 PM GMT +0000).
[18:38:01] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWlS
[18:38:02] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot 03d943311 - Bot: Update SSL cert for wiki.pyxxl.tk
[18:39:43] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.60, 6.20, 6.56
[18:40:01] <Zppix>	 paladox:  why is mw3 seem to be struggling?
[18:40:40] * paladox has been trying to find out for quite a while (months)
[18:40:49] <paladox>	 only appears to happen at this time of day
[18:41:28] <Zppix>	 paladox:  but why only mw3?
[18:41:45] <Zppix>	 paladox:  is there something on mw3 not on the other mw hosts?
[18:42:06] <Voidwalker>	 if it's at a specific time of day, I would almost think it's some sort of cron job
[18:42:13] <paladox>	 yup, the job runner
[18:42:14] <paladox>	 so it's naturally going to have a higher load then the other mw*
[18:42:18] <Voidwalker>	 either ours, or someone else's
[18:42:32] <Zppix>	 paladox:  the job runner runs jobs for all the wikis correct?
[18:42:44] <paladox>	 but it also happens to mw[12]
[18:42:46] <paladox>	 higher load
[18:42:55] <paladox>	 so i guess it could be php-fpm?
[18:43:06] <paladox>	 yup
[18:43:10] <Zppix>	 paladox:  thats my guess why is it mw3 is the only one doing job runner?
[18:43:19] <paladox>	 https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw3.miraheze.org&var-port=9100&fullscreen&panelId=281
[18:43:20] <ZppixBot>	 [ Grafana ] - grafana.miraheze.org
[18:43:28] <paladox>	 well what causes that pattern... heh
[18:43:42] <paladox>	 Zppix since we only have three servers.
[18:43:42] <Zppix>	 paladox:  influx of changes to the wikis is my guess
[18:43:53] <paladox>	 we limit the runner to 1 i think
[18:43:56] <Zppix>	 paladox:  why cant all the mw share the job runner/
[18:44:09] <paladox>	 so we basically have 2 runners (one running datadumps and another for the rest)
[18:44:19] <paladox>	 because then it'll cause high load on all mw?
[18:45:05] <Zppix>	 paladox:  but if it they all share the job runner queue and somehow distrbute the jobs evenly it should even it out?
[18:45:14] <paladox>	 Nope
[18:45:36] <paladox>	 Since it'll still be running the php script
[18:45:53] <Zppix>	 paladox:   so you can do like a "load balancer" for the job runner?
[18:45:55] <Zppix>	 cant*
[18:46:03] <paladox>	 I'm not sure
[18:46:07] <paladox>	 i haven't looked into that
[18:46:15] <paladox>	 as wmf got rid of there redis job runner
[18:46:19] <Zppix>	 paladox:  I would think if you did that you could elevate some of the load?
[18:46:37] <paladox>	 but then that'll distribute the load onto the other mw*
[18:46:44] <Voidwalker>	 lol, elevate
[18:47:10] <paladox>	 Voidwalker increase mw* resources?
[18:47:58] <Voidwalker>	 me thinks the proper word would be reduce
[18:48:04] <Zppix>	 but my guess is that around this time is when an influx of activity happens thus increasing the need for jobs
[18:48:32] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[18:48:35] <paladox>	 i doin't think this may be the job runner... at least it's running one 2 jobs
[18:49:12] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[18:49:29] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[18:49:38] <Zppix>	 paladox:  does htop show anything when icinga alerts?
[18:49:54] <paladox>	 i'll have to wait for it to alert again to say
[18:50:16] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[18:50:28] <paladox>	 i see alot of mfsmount processes
[18:50:31] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[18:50:37] <Zppix>	 paladox:  to bad htop doesnt have like a history function so you could see the stats of a certain time
[18:50:53] <paladox>	 root       111  0.0  0.0 492900   372 ?        S<sl Oct05   0:47 mfsmount /mnt/mediawiki-static -o rw,nosuid,nodev,noatime,mfsmaster=185.52.1.71,mfsport=9421,enablefilelocks=1,mfsdelayedinit,big_writes
[18:50:53] <paladox>	 root      1986  2.0  1.4 2223104 15536 ?       S<sl Oct05  81:16 mfsmount /mnt/mediawiki-static -o rw,nosuid,nodev,noatime,mfsmaster=185.52.1.71,mfsport=9421,enablefilelocks=1,mfsdelayedinit,big_writes,mfsioretries=1
[18:51:02] <paladox>	 hmm, why does it have two processes running...
[18:51:11] <Zppix>	 that looks like maybe mediawiki writing media paladox 
[18:51:16] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[18:51:27] <Zppix>	 paladox:  one has msioretries the other doesnt
[18:51:35] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 8 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[18:51:37] <paladox>	 ohhhhhhhh
[18:51:44] * paladox kills
[18:51:55] <paladox>	 i updated the config
[18:52:02] <Zppix>	 paladox:  ok?
[18:52:02] <paladox>	 so some how the change didn't take effect?
[18:52:17] <Zppix>	 paladox:  my guess is maybe puppet or something didnt restart the proc?
[18:52:25] <Zppix>	 or kill it or whatever
[18:52:43] <paladox>	 oh, mount
[18:52:44] <paladox>	 ignore the last sentence :)
[18:52:51] <paladox>	 !log depool mw3
[18:53:03] * Zppix is now confused
[18:53:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:53:50] <paladox>	 ignore the "oh, mount"
[18:53:55] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[18:54:21] <Zppix>	 paladox:  so what was the issue with the 2 procs?
[18:54:42] <paladox>	 I just think that it started another process when i changed the config
[18:54:52] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[18:54:54] <Zppix>	 paladox:  should it of auto killed the other?
[18:54:57] <Zppix>	 shouldnt*
[18:55:05] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[18:55:09] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[18:55:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW8e
[18:55:12] <paladox>	 i would have thought so when i did `umount mediawiki-static`
[18:55:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03ff15ff2 - BOT: Updating services config for wikis
[18:55:32] <Zppix>	 paladox:  maybe the command error and you didnt notice or something
[18:55:50] <paladox>	 maybe
[18:56:04] <Zppix>	 paladox:  but i still doubt those 2 procs had to do with it
[18:56:06] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[18:56:11] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[18:56:14] <paladox>	 !log kill -9 111 - mw3
[18:56:18] <Zppix>	 paladox:  considering one of them would end up canceling the other one out anyway
[18:56:33] <paladox>	 !log repool mw3
[18:56:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:56:51] <paladox>	 well this still needs investigating
[18:56:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:56:56] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[18:57:45] <Zppix>	 paladox:  well try not to blow up a datacenter, or something xD
[18:57:51] <paladox>	 lol
[18:58:14] <Zppix>	 paladox:  i mean we had to repaint the red button pink hoping that would stop you pressing it so..
[18:58:22] <paladox>	 heh
[18:58:38] * paladox gives pink button to Voidwalker as a present :P
[18:59:11] <Voidwalker>	 hmmmmmmmm
[18:59:56] <Zppix>	 Voidwalker:  well atleast he cant press it if you have it xD
[19:00:13] <Voidwalker>	 yeah, better get a glass case for it just in case
[19:00:24] <paladox>	 lol
[19:01:33] * paladox wonders if he should lower mfswriteworkers from 10 to 5
[19:01:36] <paladox>	 (workers)
[19:01:39] <Zppix>	 Voidwalker:  glass case wouldnt stop paladox, he would just put a server by it and make the server explode
[19:01:46] <paladox>	 lol
[19:01:51] <Zppix>	 paladox:  i dont even know what those do so i couldnt tell you either way lol
[19:02:11] <paladox>	 per description: "define number of write workers (default: 10)"
[19:03:18] <paladox>	 !log hack fstab on mw3 to test some configuation changes
[19:03:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:03:50] <paladox>	 !log repool mw3
[19:03:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:08:43] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures
[19:09:42] <Not-f817>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[19:09:44] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeW83
[19:09:45] <Not-f817>	 [02miraheze/puppet] 07paladox 03264d6b8 - jobrunner: Move GUP into basic
[19:09:47] <Not-f817>	 [02puppet] 07paladox opened pull request 03#1094: jobrunner: Move GUP into basic - 13https://git.io/JeW8s
[19:11:20] <Not-f817>	 [02puppet] 07paladox closed pull request 03#1094: jobrunner: Move GUP into basic - 13https://git.io/JeW8s
[19:11:21] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW8Z
[19:11:23] <Not-f817>	 [02miraheze/puppet] 07paladox 037edfd9b - jobrunner: Move GUP into basic (#1094)
[19:13:27] <paladox>	 !log depool mw3 again
[19:14:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:19:31] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW8B
[19:19:32] <Not-f817>	 [02miraheze/puppet] 07paladox 03dcd3317 - mediawiki: Set nginx timeout to 12s for /php_status
[19:29:06] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[19:34:12] <paladox>	 !log restarted nginx on mw3
[19:37:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:37:50] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3
[19:38:23] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[19:38:51] <Not-f817>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW8b
[19:38:53] <Not-f817>	 [02miraheze/ssl] 07paladox 03864b156 - Update wiki.pyxxl.tk.crt
[19:39:39] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax]
[19:39:40] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[19:40:07] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:40:07] <icinga-miraheze>	 PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[19:40:08] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host
[19:40:09] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[19:40:10] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[19:40:11] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host
[19:40:57] <icinga-miraheze>	 PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host
[19:41:18] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[19:42:08] <paladox>	 huh
[19:42:55] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[19:42:55] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[19:43:17] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[19:43:19] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 0.667 second response time
[19:43:20] <icinga-miraheze>	 RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.006 second response time
[19:43:21] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.006 second response time
[19:43:52] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[19:43:59] <icinga-miraheze>	 RECOVERY - wiki.pyxxl.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pyxxl.tk' will expire on Mon 06 Jan 2020 06:37:25 PM GMT +0000.
[19:44:00] <icinga-miraheze>	 RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.390 second response time
[19:44:16] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[19:44:23] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[19:46:22] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:46:22] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[19:47:19] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:47:23] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW4e
[19:47:24] <Not-f817>	 [02miraheze/puppet] 07paladox 039907cb5 - Update prometheus-php-fpm.systemd
[19:50:26] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 1.276 second response time
[19:52:09] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[19:52:24] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[19:55:21] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures
[19:57:16] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeW4n
[19:57:18] <Not-f817>	 [02miraheze/puppet] 07paladox 03eaabc11 - Revert "Update prometheus-php-fpm.systemd" This reverts commit 9907cb5bdeb699749896eedf2088bc5f7bf50fd4.
[19:57:24] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[20:04:47] <RhinosF1>	 .mhca ETTWiki
[20:04:47] <ZppixBot>	 https://meta.miraheze.org/wiki/Special:CentralAuth/ETTWiki
[20:05:13] <RhinosF1>	 Zppix: can you issue a UPOL warning
[20:05:30] <RhinosF1>	 .mh Username_Policy
[20:05:31] <ZppixBot>	 https://meta.miraheze.org/wiki/Username_Policy
[20:11:00] <Zppix>	 {{done}}
[20:14:33] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[20:14:53] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3
[20:15:21] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[20:17:48] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[20:17:58] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[20:18:06] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[20:42:54] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.73, 1.55, 1.01
[20:45:31] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.96, 1.27, 0.98
[20:54:25] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:55:17] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[20:57:04] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.291 second response time
[20:57:53] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[21:13:17] <Voidwalker>	 Zppix, RhinosF1, regarding the UPOL warning, the policy states "exact name of a wiki", which wiki is this referring to?
[21:13:40] <Zppix>	 Voidwalker:  i am just doing what i was told 
[21:13:47] <Zppix>	 so thats RhinosF1's question
[21:13:50] <RhinosF1>	 hmm, I may have missed that
[21:14:06] <RhinosF1>	 Zppix: you're supposed to double check
[21:14:09] <Examknow>	 RhinosF1: Why did you ban me from your channel?
[21:14:22] <RhinosF1>	 Examknow: take it to #quirc-bans
[21:14:53] <RhinosF1>	 Zppix: would you jump off a cliff I told you to
[21:15:12] <Zppix>	 RhinosF1:  no im not stupid
[21:15:36] <JohnLewis>	 Zppix: you are :P
[21:15:55] <Zppix>	 JohnLewis:  Thanks :P\
[21:16:18] <paladox>	 !log sudo service prometheus-node-exporter stop - mw3
[21:16:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:24:34] <paladox>	 !log sudo service prometheus-node-exporter start - mw3
[21:24:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:25:48] <Examknow>	 Zppix: Is the bot that you host on heroku made with sopel?
[21:31:17] <RhinosF1>	 Examknow: i don't think MH-Discord is
[21:31:19] <RhinosF1>	 .sourc
[21:31:22] <RhinosF1>	 .source
[21:31:23] <ZppixBot>	 RhinosF1: My code can be found here: https://github.com/Pix1234/ZppixBot-Source
[21:31:45] <RhinosF1>	 Examknow: https://github.com/Pix1234/discord-irc
[21:31:47] <ZppixBot>	 [ GitHub - Pix1234/discord-irc: Connects Discord and IRC channels by sending messages back and forth. ] - github.com
[21:32:57] <Zppix>	 RhinosF1:  no
[21:33:00] <Zppix>	 Examknow:  ^
[21:33:06] <Zppix>	 Examknow:  its in nodejs
[21:33:14] <Examknow>	 k
[21:33:30] <Zppix>	 why?
[21:35:42] <paladox>	 !log apply fstab hack to mw1 (depool and repool)
[21:36:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:40:24] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures
[22:06:38] <Not-f817>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeWRu
[22:06:40] <Not-f817>	 [02miraheze/puppet] 07paladox 0393d48a4 - mediawiki: Set mfswriteworkers to 100 for lizardfs client
[22:07:19] <paladox>	 !log depool and repool mw2
[22:07:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:08:11] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:13:46] <paladox>	 !log removed hacks from mw*, puppet ran.
[22:13:49] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[22:14:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:36:50] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[22:37:03] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1
[22:37:14] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[22:37:14] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1
[22:39:23] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[22:39:33] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[22:39:42] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[22:39:42] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy