[02:17:24] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%);
[02:24:07] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.02, 1.95, 1.17
[02:26:06] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.35, 1.70, 1.18
[02:35:06] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 6.76, 3.82, 2.17
[03:26:03] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.42, 0.75, 1.87
[03:30:03] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.58, 0.65, 1.57
[06:25:27] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2916 MB (12% inode=94%);
[08:04:41] <BIPU>	 Hello, yesterday I reported an error accessing one of my pages on the Altverse wiki and the problem persists today. Please, could you tell me if it is possible to solve it? Thank you. The error message is: [e902dad1c0606fe48c9098cd] 2020-03-10 08:04:26: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"
[08:04:56] <BIPU>	 And the page is: https://wiki.conworlds.org/wiki/Storting
[08:04:57] <ZppixBot>	 [ Internal error - Constructed Worlds ] - wiki.conworlds.org
[11:16:14] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu]
[11:22:16] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[12:03:14] <BIPU>	 Problems continue. Now I cant log in in Altverse
[12:07:20] <BIPU>	 I asked for a new temporary password and when I tried to write a new one the same error message appears
[12:14:46] <BIPU>	 I'm sorry, but is it supposed that this channel is for support? Is there any other place to get it? Thanks
[12:22:55] <paladox>	 BIPU: hi 
[12:22:56] <paladox>	 BIPU: I mentioned yesterday that the revision was missing (and pinged SPF|Cloud) :)
[12:22:57] <paladox>	 BIPU: does logging in using private mode work? What error do you get?
[12:25:24] <BIPU>	 The logging message is: Centralized login. The authentication signal provided has expired or is invalid.
[12:26:46] <BIPU>	 I tried to use the "I dont remember the password" and received a temporary one by mail. When I tried to log in and I use the temporary one, it seems to work and ask me for a new one. When I write a new password the error message appears again.
[12:27:43] <paladox>	 BIPU: does clicking return work?
[12:27:44] <paladox>	 Yeh, I’ve seen this before, click return *should* work
[12:28:44] <BIPU>	 It doesnt work. It worked some time in the pass, but not today
[12:29:01] <BIPU>	 Sorry... some time in the past
[12:30:18] <BIPU>	 Now when clicking return the browser ask for confirming form forwarding and when I do it nothing happens
[12:31:42] <BIPU>	 And appears this error: ERR_CACHE_MISS
[12:33:47] <BIPU>	 I tried with another browser and is the same. When cliking return the page shows me again the log in dialogue
[12:34:15] <paladox>	 Hmm
[12:34:19] <paladox>	 I’m currently mobile so cannot look into that atm :(
[12:35:00] <BIPU>	 Ok, thankyoe. If you were able to have a look later I will thank you :-)
[12:36:33] <paladox>	 Sure, I guess your from the US? (Trying to work out which cache proxy your going through)
[12:37:50] <BIPU>	 No, I'm from Spain
[12:39:15] <paladox>	 Oh
[13:35:45] <Examknow>	 paladox: is the wc issue fixed yet?
[13:37:51] <paladox>	 Nope
[13:38:18] <Examknow>	 oh
[13:38:46] <Examknow>	 we might want to post a centralnotice bcs there are many users waiting on their wikis to be created
[14:27:59] <BIPU>	 Paladox: Tocomplement my prior inforation, I have tried to create a new user and it is also impossible. The error now is: Error 503 Backend fetch failed, forwarded for 81.39.221.102, 127.0.0.1
[14:29:30] <Voidwalker>	 503s should only be temporary
[14:30:25] <Voidwalker>	 if it occurs every time you attempt to create a new account, then that's pretty serious, but probably unrelated to the login failures
[14:30:57] <BIPU>	 Ok, It was only as info 
[14:33:37] <BIPU>	 Hey.... I've been able to login now!!!
[15:03:46] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 51%
[15:07:45] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 61%
[15:09:45] <icinga-miraheze>	 RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 35%
[15:18:17] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 59%
[15:20:20] <icinga-miraheze>	 RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 30%
[16:37:05] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw2
[16:38:12] <RhinosF1>	 Zppix: ^ whois that host from BIPU pls for me
[16:39:02] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy
[16:52:32] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvKBu
[16:52:34] <Not-a789>	 [02miraheze/mw-config] 07paladox 03c9ae67d - Update LocalSettings.php
[16:52:35] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3
[16:52:37] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2941: Update LocalSettings.php - 13https://git.io/JvKBz
[16:52:56] <Not-a789>	 [02mw-config] 07paladox edited pull request 03#2941: Add cp ipv6 address and cp[67] back to wgCdnServers - 13https://git.io/JvKBz
[16:54:05] <travis-ci>	 miraheze/mw-config/paladox-patch-3/c9ae67d - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/660713382
[16:55:48] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKBQ
[16:55:50] <Not-a789>	 [02miraheze/puppet] 07paladox 036e8ce15 - nginx: Remove cp2
[16:56:27] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKB7
[16:56:28] <Not-a789>	 [02miraheze/puppet] 07paladox 03cd21de3 - nginx: Remove cp4
[16:56:37] <RhinosF1>	 paladox: can you update the on-wiki migration checklist
[16:56:52] <paladox>	 Why?
[16:57:08] <RhinosF1>	 paladox: because you're removing cps
[16:57:40] <paladox>	 I mean, i've forgotten where this is secondly i'm not following that.
[16:58:03] <Not-a789>	 [02mw-config] 07paladox synchronize pull request 03#2941: Add cp ipv6 address and cp[67] back to wgCdnServers - 13https://git.io/JvKBz
[16:58:04] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvKBh
[16:58:06] <Not-a789>	 [02miraheze/mw-config] 07paladox 0352c6f79 - Update LocalSettings.php
[16:58:27] <Not-a789>	 [02mw-config] 07paladox edited pull request 03#2941: Add cp[67] back to wgCdnServers and remove cp4 - 13https://git.io/JvKBz
[16:59:12] <RhinosF1>	 paladox: https://meta.miraheze.org/wiki/Tech:Cloud_Server_Comparison
[16:59:12] <ZppixBot>	 [ Tech:Cloud Server Comparison - Miraheze Meta ] - meta.miraheze.org
[16:59:43] <paladox>	 Ok, what am i suppose to update on this? Secondly you can do that, you doin't need me too :)
[17:00:12] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2941: Add cp[67] back to wgCdnServers and remove cp4 - 13https://git.io/JvKBz
[17:00:13] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKRT
[17:00:15] <Not-a789>	 [02miraheze/mw-config] 07paladox 039f69981 - Add cp[67] back to wgCdnServers and remove cp4 (#2941) * Update LocalSettings.php * Update LocalSettings.php
[17:00:16] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3
[17:00:18] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3
[17:00:27] <RhinosF1>	 The migrated column, just tick off what's migrated or have both running. If it's only [67] replacing 4 then I can
[17:00:36] <RhinosF1>	 If not, I don't know what else is in use
[17:00:45] <RhinosF1>	 that isn't updated
[17:00:51] <paladox>	 ok
[18:07:22] <hispano76>	 hi, I have this problem in my wiki https://www.mediawiki.org/wiki/Extension_talk:Cargo/Archive_August_to_September_2018#Cargo_not_updating_tables%2C_nothing_happening.It seems to require the use of cargoRecreateData.php, or refreshLinks.php  (in Hispanowiki)    
[18:07:23] <ZppixBot>	 [ Extension talk:Cargo/Archive August to September 2018 - MediaWiki ] - www.mediawiki.org
[18:09:26] <paladox>	 RhinosF1 done.
[18:13:20] <Zppix>	 hispano76:  will do
[18:14:59] <Zppix>	 hispano76:  i just ran cargoRecreateData did that fix
[18:17:00] <Zppix>	 !log run cargoRecreateData.php and refreshLinks.php on jobrunner1 on hispanowiki per user request on IRC
[18:17:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:21:07] <RhinosF1>	 paladox: you’re a star, thx!
[18:43:22] <icinga-miraheze>	 PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lab612.at' expires in 15 day(s) (Thu 26 Mar 2020 06:40:57 PM GMT +0000).
[18:43:57] <icinga-miraheze>	 PROBLEM - wiki.exnihilolinux.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.exnihilolinux.org' expires in 15 day(s) (Thu 26 Mar 2020 06:40:25 PM GMT +0000).
[18:46:36] <icinga-miraheze>	 PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Thu 26 Mar 2020 06:44:34 PM GMT +0000).
[18:47:27] <icinga-miraheze>	 PROBLEM - nenawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nenawiki.org' expires in 15 day(s) (Thu 26 Mar 2020 06:44:34 PM GMT +0000).
[18:50:01] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKuK
[18:50:03] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot 0305b05d6 - Bot: Update SSL cert for www.lab612.at
[18:50:44] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKuP
[18:50:45] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot 033686dbe - Bot: Update SSL cert for wiki.exnihilolinux.org
[18:52:40] <paladox>	 !log set features.read-only to on, on lizardfs6
[18:52:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:54:00] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKuX
[18:54:02] <Not-a789>	 [02miraheze/mw-config] 07paladox 039253a05 - Update Sitenotice.php
[18:54:10] <Zppix>	 i was just about to do that paladox
[18:54:23] <paladox>	 heh
[18:54:42] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKu1
[18:54:44] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot 03dc451ec - Bot: Update SSL cert for nenawiki.org
[18:57:20] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKuS
[18:57:22] <Not-a789>	 [02miraheze/ssl] 07MirahezeSSLBot 030465120 - Bot: Update SSL cert for ndg.nenawiki.org
[18:59:52] <hispano76>	 hum,  https://hispano.miraheze.org/wiki/Especial:CargoTables
[18:59:53] <ZppixBot>	 [ Tablas de Cargo - Hispano ] - hispano.miraheze.org
[19:01:33] <Zppix>	 hispano76:  refreshlinks.php is still running
[19:02:03] <hispano76>	 ah, ok
[19:03:01] <Zppix>	 approx 2500 out of 34886 links hispano76 
[19:03:08] <Zppix>	 s/links/pages
[19:03:08] <ZppixBot>	 Zppix meant to say: approx 2500 out of 34886 pages hispano76 
[19:03:29] <icinga-miraheze>	 RECOVERY - nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Mon 08 Jun 2020 05:54:36 PM GMT +0000.
[19:03:34] <icinga-miraheze>	 RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Mon 08 Jun 2020 05:49:55 PM GMT +0000.
[19:03:52] <icinga-miraheze>	 RECOVERY - wiki.exnihilolinux.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.exnihilolinux.org' will expire on Mon 08 Jun 2020 05:50:37 PM GMT +0000.
[19:03:56] <hispano76>	 ok
[19:04:03] <icinga-miraheze>	 RECOVERY - ndg.nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Mon 08 Jun 2020 05:54:36 PM GMT +0000.
[19:04:04] <hispano76>	 thanks, thanks ZppixBot 
[19:04:09] <Zppix>	 np
[19:05:44] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[19:06:38] <Not-a789>	 [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvKzf
[19:06:40] <Not-a789>	 [02miraheze/MatomoAnalytics] 07paladox 031d13da7 - Do not fail hard if you have issues with Matomo Log a message instead.
[19:06:41] <Not-a789>	 [02MatomoAnalytics] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fN4LT
[19:06:43] <Not-a789>	 [02MatomoAnalytics] 07paladox opened pull request 03#19: Do not fail hard if you have issues with Matomo - 13https://git.io/JvKzJ
[19:09:20] <travis-ci>	 miraheze/MatomoAnalytics/paladox-patch-2/1d13da7 - paladox The build was broken. https://travis-ci.org/miraheze/MatomoAnalytics/builds/660766798
[19:09:53] <Not-a789>	 [02MatomoAnalytics] 07paladox synchronize pull request 03#19: Do not fail hard if you have issues with Matomo - 13https://git.io/JvKzJ
[19:09:55] <Not-a789>	 [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvKzk
[19:09:56] <Not-a789>	 [02miraheze/MatomoAnalytics] 07paladox 038f14c49 - Update MatomoAnalytics.php
[19:12:32] <travis-ci>	 miraheze/MatomoAnalytics/paladox-patch-2/8f14c49 - paladox The build was fixed. https://travis-ci.org/miraheze/MatomoAnalytics/builds/660768137
[19:53:33] <Zppix>	 !log update cw_requests set cw_status = 'approved' where cw_id = 10988;
[19:53:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:58:25] <Not-a789>	 [02MatomoAnalytics] 07paladox closed pull request 03#19: Do not fail hard if you have issues with Matomo - 13https://git.io/JvKzJ
[19:58:27] <Not-a789>	 [02MatomoAnalytics] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fN4LT
[19:58:28] <Not-a789>	 [02miraheze/MatomoAnalytics] 07paladox deleted branch 03paladox-patch-2
[20:01:53] <Not-a789>	 [02mw-config] 07Hispano76 opened pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvKgX
[20:31:56] <paladox>	 !log MariaDB [mhglobal]> delete from matomo; - db4
[20:32:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:32:09] <paladox>	 !log root@jobrunner1:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/addMatomo.php
[20:32:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:48:46] <paladox>	 !log delete *matomo* key in redis
[20:48:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:03:07] <paladox>	 !log restart jobrunner + jobchron on jobrunner1
[21:03:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:07:05] <Zppix>	 hispano76: what cargo tables do you have on hispanowiki?
[21:07:11] <Zppix>	 are supposed to be*
[21:08:45] <hispano76>	 Zppix:  https://hispano.miraheze.org/wiki/Especial:CargoTables It used to be two templates that called Cargo but I removed the code to do a parameter reorganization. Since then I got the message
[21:08:46] <ZppixBot>	 [ Tablas de Cargo - Hispano ] - hispano.miraheze.org
[21:09:31] <Zppix>	 hispano76: so should it be blank or should there be tables?
[21:11:15] <hispano76>	 Zppix: hehe That's already been arranged as far as I can see, I don't see the message.   :P 
[21:11:34] <Zppix>	 ok
[21:11:37] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKaq
[21:11:38] <Not-a789>	 [02miraheze/puppet] 07paladox 03d5cfb4a - jobrunner: Use new jobrunner redis if using new servers
[21:11:40] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[21:11:41] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1283: jobrunner: Use new jobrunner redis if using new servers - 13https://git.io/JvKam
[21:11:46] <hispano76>	 thanks Zppix 
[21:11:48] <hispano76>	 :)
[21:11:52] <Zppix>	 np
[21:13:15] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKaO
[21:13:16] <Not-a789>	 [02miraheze/puppet] 07paladox 03add8309 - Update jobrunner.json.erb
[21:13:18] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1283: jobrunner: Use new jobrunner redis if using new servers - 13https://git.io/JvKam
[21:13:28] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1283: jobrunner: Use new jobrunner redis if using new servers - 13https://git.io/JvKam
[21:13:30] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvKaG
[21:13:31] <Not-a789>	 [02miraheze/puppet] 07paladox 03bd4434e - jobrunner: Use new jobrunner redis if using new servers (#1283) * jobrunner: Use new jobrunner redis if using new servers * Update jobrunner.json.erb
[21:13:33] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[21:13:34] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[21:31:07] <JohnLewis>	 why do we even bother maintaining mw3 as a jobrunner currently?
[21:31:50] <paladox>	 lizardfs6 is the jobrunner
[21:31:59] <paladox>	 JohnLewis that's only used on the new servers
[21:32:04] <paladox>	 the old infra still uses lizardfs6
[21:32:22] <JohnLewis>	 okay, that makes even less sense
[21:32:32] <JohnLewis>	 lizardfs6 is in London, jobrunner1 is in London
[21:32:38] <JohnLewis>	 -> use jobrunner1
[21:33:17] <paladox>	 !log root@lizardfs6:/home/paladox# gluster volume set mvol features.read-only off
[21:33:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:33:28] <paladox>	 ok
[21:34:11] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKai
[21:34:13] <Not-a789>	 [02miraheze/puppet] 07paladox 0352978a4 - jobrunner: Use jobrunner1 for old infra too
[21:34:14] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[21:34:16] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1284: jobrunner: Use jobrunner1 for old infra too - 13https://git.io/JvKaP
[21:35:04] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKaX
[21:35:05] <Not-a789>	 [02miraheze/mw-config] 07paladox 03f9e9d28 - Switch to using jobrunner1 on old infra
[21:35:20] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKa1
[21:35:22] <Not-a789>	 [02miraheze/puppet] 07paladox 031cac404 - Update jobrunner.pp
[21:35:23] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1284: jobrunner: Use jobrunner1 for old infra too - 13https://git.io/JvKaP
[21:36:04] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKaM
[21:36:06] <Not-a789>	 [02miraheze/puppet] 07paladox 033193dbf - Update lizardfs6.yaml
[21:36:07] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1284: jobrunner: Use jobrunner1 for old infra too - 13https://git.io/JvKaP
[21:36:17] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1284: jobrunner: Use jobrunner1 for old infra too - 13https://git.io/JvKaP
[21:36:18] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvKaD
[21:36:20] <Not-a789>	 [02miraheze/puppet] 07paladox 031ec7914 - jobrunner: Use jobrunner1 for old infra too (#1284)
[21:36:21] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[21:36:23] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[21:39:43] <icinga-miraheze>	 PROBLEM - lizardfs6 JobRunner Service on lizardfs6 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService'
[21:39:50] <icinga-miraheze>	 PROBLEM - lizardfs6 JobChron Service on lizardfs6 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService'
[21:46:22] <paladox>	 !log MariaDB [browndustwiki]> INSERT INTO slot_roles (`role_id`,`role_name`) VALUES (1,'main'); - db4
[21:46:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:50:16] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKVm
[21:50:17] <Not-a789>	 [02miraheze/mw-config] 07paladox 03b6cab4c - Disable the site notice
[21:58:24] <paladox>	 !log MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (1, 'wikitext'); -db4
[21:58:38] <paladox>	 !log MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (5, 'json'); -db4
[21:58:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:58:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:58:55] <paladox>	 !log MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (3, 'javascript'); -db4
[21:59:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:59:07] <paladox>	 !log MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (2, 'css'); -db4
[21:59:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:59:22] <paladox>	 !log MariaDB [browndustwiki]> INSERT INTO content_models (`model_id`,`model_name`) VALUES (4, 'Scribunto'); -db4
[21:59:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:16:05] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKVw
[22:16:06] <Not-a789>	 [02miraheze/puppet] 07paladox 03828b0df - redis: Enable RDB on the jobrunner only
[22:16:08] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[22:16:09] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1285: redis: Enable RDB on the jobrunner only - 13https://git.io/JvKVr
[22:29:42] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKVd
[22:29:44] <Not-a789>	 [02miraheze/puppet] 07paladox 03200dbea - set default puppetserver to puppet2
[22:34:04] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures
[22:44:52] <paladox>	 !log shutdown puppet1
[22:44:55] <paladox>	 !log shutdown cp4
[22:45:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:45:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:45:29] <paladox>	 !log shutdown misc4
[22:45:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:47:32] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKwV
[22:47:33] <Not-a789>	 [02miraheze/puppet] 07paladox 035e67896 - misc1: Remove icinga2
[22:49:11] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKwo
[22:49:13] <Not-a789>	 [02miraheze/dns] 07paladox 037de92ab - Set icing backend as mon1
[22:49:32] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKw6
[22:49:33] <Not-a789>	 [02miraheze/dns] 07paladox 036dd5d17 - Update miraheze.org
[22:53:10] <paladox>	 !log apt-get upgrade - mon1 (icinga2/puppet-agent)
[22:53:10] <icinga-miraheze>	 PROBLEM - lizardfs6 GlusterFS port 24007 on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:53:11] <icinga-miraheze>	 PROBLEM - db4 MySQL on db4 is CRITICAL: Access denied for user 'icinga'@'51.89.160.138' (using password: YES)
[22:53:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:53:42] <icinga-miraheze>	 PROBLEM - lizardfs6 GlusterFS port 49152 on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:54:23] <icinga-miraheze>	 PROBLEM - db6 Puppet on db6 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 days ago with 0 failures
[22:55:28] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKwH
[22:55:29] <Not-a789>	 [02miraheze/puppet] 07paladox 0338c161c - Update firewall.yaml
[22:55:55] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[22:55:56] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[22:56:05] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[22:57:05] <Zppix>	 why is puppet on db6 disabled paladox
[22:57:23] <paladox>	 Because of the issue on sunday
[22:58:00] <icinga-miraheze>	 PROBLEM - gluster1 Disk Space on gluster1 is CRITICAL: DISK CRITICAL - free space: / 36373 MB (5% inode=82%);
[22:59:54] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[23:00:16] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for test2 on cp8 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.004 second response time
[23:00:35] <icinga-miraheze>	 PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 248.11 ms
[23:01:31] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for test2 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.002 second response time
[23:01:41] <icinga-miraheze>	 PROBLEM - db6 MySQL on db6 is CRITICAL: Can't connect to MySQL server on '51.89.160.130' (115)
[23:01:51] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for lizardfs6 on cp8 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8083: HTTP/1.1 200 OK
[23:02:12] <icinga-miraheze>	 RECOVERY - gluster1 GlusterFS port 49152 on gluster1 is OK: TCP OK - 0.000 second response time on 51.77.107.209 port 49152
[23:02:46] <icinga-miraheze>	 RECOVERY - lizardfs6 GlusterFS port 49152 on lizardfs6 is OK: TCP OK - 0.000 second response time on 54.36.165.161 port 49152
[23:02:48] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for lizardfs6 on cp3 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8083: HTTP/1.1 200 OK
[23:03:14] <icinga-miraheze>	 RECOVERY - gluster1 GlusterFS port 24007 on gluster1 is OK: TCP OK - 0.000 second response time on 51.77.107.209 port 24007
[23:03:19] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKrt
[23:03:20] <Not-a789>	 [02miraheze/puppet] 07paladox 03a0bd5a8 - services: Add mon1 to firewall
[23:04:13] <icinga-miraheze>	 RECOVERY - lizardfs6 GlusterFS port 24007 on lizardfs6 is OK: TCP OK - 0.000 second response time on 54.36.165.161 port 24007
[23:04:18] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[23:05:04] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:05:10] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKrY
[23:05:11] <Not-a789>	 [02miraheze/puppet] 07paladox 032f27b85 - puppetdb: Add mon1 to firewall
[23:05:17] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[23:05:18] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[23:07:14] <icinga-miraheze>	 RECOVERY - services2 zotero on services2 is OK: TCP OK - 0.001 second response time on 51.89.160.141 port 1969
[23:07:17] <icinga-miraheze>	 RECOVERY - services2 restbase on services2 is OK: TCP OK - 0.000 second response time on 51.89.160.141 port 7231
[23:07:21] <icinga-miraheze>	 RECOVERY - services1 citoid on services1 is OK: TCP OK - 0.000 second response time on 51.89.160.132 port 6927
[23:07:34] <icinga-miraheze>	 RECOVERY - services1 restbase on services1 is OK: TCP OK - 0.000 second response time on 51.89.160.132 port 7231
[23:07:49] <icinga-miraheze>	 RECOVERY - services1 parsoid on services1 is OK: TCP OK - 0.000 second response time on 51.89.160.132 port 8142
[23:07:55] <icinga-miraheze>	 RECOVERY - services2 parsoid on services2 is OK: TCP OK - 0.000 second response time on 51.89.160.141 port 8142
[23:08:23] <icinga-miraheze>	 RECOVERY - services1 zotero on services1 is OK: TCP OK - 0.000 second response time on 51.89.160.132 port 1969
[23:08:43] <icinga-miraheze>	 RECOVERY - services2 citoid on services2 is OK: TCP OK - 0.000 second response time on 51.89.160.141 port 6927
[23:09:15] <icinga-miraheze>	 RECOVERY - services1 electron on services1 is OK: TCP OK - 0.000 second response time on 51.89.160.132 port 3000
[23:09:16] <icinga-miraheze>	 RECOVERY - services2 electron on services2 is OK: TCP OK - 0.000 second response time on 51.89.160.141 port 3000
[23:10:04] <icinga-miraheze>	 RECOVERY - puppet2 puppetdb on puppet2 is OK: TCP OK - 0.013 second response time on 51.89.160.129 port 8081
[23:11:06] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKrn
[23:11:07] <Not-a789>	 [02miraheze/puppet] 07paladox 037abb81f - Update default-grants.sql.erb
[23:18:10] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKry
[23:18:11] <Not-a789>	 [02miraheze/puppet] 07paladox 03489202d - Update init.pp
[23:18:13] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1285: redis: Enable RDB on the jobrunner only - 13https://git.io/JvKVr
[23:18:31] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1285: redis: Enable RDB on the jobrunner only - 13https://git.io/JvKVr
[23:18:33] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvKrH
[23:18:34] <Not-a789>	 [02miraheze/puppet] 07paladox 03e4a12b9 - redis: Enable RDB on the jobrunner only (#1285) * redis: Enable RDB on the jobrunner only * Update init.pp
[23:18:36] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[23:18:37] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[23:19:17] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKr5
[23:19:18] <Not-a789>	 [02miraheze/puppet] 07paladox 03f22b8a0 - Set default salt master to puppet2
[23:47:06] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%);