[00:02:43] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3721 MB (19% inode=93%);
[00:03:18] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 33.89, 27.87, 22.20
[00:04:27] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 15.85, 21.04, 18.32
[00:05:20] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.82, 23.66, 21.37
[00:06:28] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.37, 19.01, 17.93
[00:11:34] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.58, 24.10, 22.11
[00:15:42] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.59, 23.79, 22.74
[00:23:11] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 32.24, 22.38, 18.56
[00:24:06] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 34.89, 26.46, 23.92
[00:25:14] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 13.89, 18.87, 17.75
[00:28:22] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.09, 22.54, 23.06
[00:32:55] <icinga-miraheze>	 PROBLEM - miraheze.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.tk' expires in 15 day(s) (Sat 11 Jul 2020 00:30:20 GMT +0000).
[00:33:40] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.87, 21.67, 19.32
[00:34:42] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.72, 22.64, 22.64
[00:35:44] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.86, 20.27, 19.19
[00:36:38] <icinga-miraheze>	 PROBLEM - miraheze.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ml' expires in 15 day(s) (Sat 11 Jul 2020 00:32:41 GMT +0000).
[00:36:46] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.08, 23.07, 22.87
[00:37:13] <icinga-miraheze>	 PROBLEM - miraheze.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.ga' expires in 15 day(s) (Sat 11 Jul 2020 00:33:53 GMT +0000).
[00:38:20] <icinga-miraheze>	 PROBLEM - miraheze.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.gq' expires in 15 day(s) (Sat 11 Jul 2020 00:35:17 GMT +0000).
[00:38:54] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.69, 22.62, 22.62
[00:41:01] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.54, 21.44, 22.16
[00:42:02] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.01, 20.77, 19.73
[00:43:06] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 32.99, 26.19, 23.81
[00:44:04] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.67, 17.85, 18.80
[00:47:19] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.54, 23.59, 23.60
[00:51:31] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.79, 22.94, 23.08
[00:54:36] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.55, 22.22, 22.81
[00:55:38] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.25, 23.94, 23.35
[00:57:40] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.01, 22.37, 22.84
[01:06:03] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.53, 22.74, 22.29
[01:08:05] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.66, 20.25, 21.48
[01:12:14] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.39, 18.31, 20.30
[01:13:33] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.87, 21.40, 16.95
[01:14:03] <icinga-miraheze>	 RECOVERY - db6 MySQL on db6 is OK: Uptime: 268 Threads: 15 Questions: 3345 Slow queries: 30 Opens: 2505 Flush tables: 1 Open tables: 2365 Queries per second avg: 12.481
[01:15:29] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.67, 18.17, 16.30
[01:19:42] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfhlM
[01:19:43] <Not-f378>	 [02miraheze/puppet] 07paladox 036cb822f - db: add db6 to firewall
[01:35:13] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfh83
[01:35:14] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 033636603 - BOT: Updating services config for wikis
[01:57:42] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out
[01:59:39] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (2.5 weeks ago)
[02:35:12] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfhBu
[02:35:14] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 03b1bd7bc - BOT: Updating services config for wikis
[02:46:58] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: connect to address 168.235.111.29 port 5666: Connection timed outconnect to host 168.235.111.29 port 5666: Connection timed out
[02:48:59] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (2.5 weeks ago)
[03:25:59] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.74, 1.70, 1.23
[03:27:55] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.59, 1.29, 1.13
[05:28:27] <Not-f378>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JfhuC
[05:28:28] <Not-f378>	 [02miraheze/ssl] 07Reception123 0329ad8c7 - re-add wiki.counterculturelabs.org cert
[06:43:57] <Not-f378>	 [02ssl] 07RhinosF1 closed pull request 03#325: [DNM] - en.famepedia.co.in - 13https://git.io/JfbDH
[07:10:23] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfh2l
[07:10:24] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 03604f1e8 - BOT: Updating services config for wikis
[08:30:25] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfhw9
[08:30:27] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 0388443e8 - BOT: Updating services config for wikis
[09:00:25] <Not-f378>	 [02miraheze/mediawiki] 07RhinosF1 created branch 03update-extskin 13https://git.io/Jfho4
[09:00:27] <Not-f378>	 [02mediawiki] 07RhinosF1 created branch 03update-extskin - 13https://git.io/vbL5b
[09:05:30] <Not-f378>	 [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03update-extskin [+0/-0/±1] 13https://git.io/Jfho2
[09:05:32] <Not-f378>	 [02miraheze/mediawiki] 07RhinosF1 03ab2b6e6 - Update Citizen
[09:06:21] <RhinosF1>	 Reception123: ^ i18n change included
[09:07:01] <Not-f378>	 [02mediawiki] 07RhinosF1 opened pull request 03#147: Update Citizen - 13https://git.io/JfhoV
[09:07:55] <Reception123>	 RhinosF1: ok then rebuild is needed on all mws and jobrunner
[09:08:12] <RhinosF1>	 yep
[09:08:20] <Not-f378>	 [02mediawiki] 07RhinosF1 closed pull request 03#147: Update Citizen - 13https://git.io/JfhoV
[09:08:21] <Not-f378>	 [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jfho1
[09:08:23] <Not-f378>	 [02miraheze/mediawiki] 07RhinosF1 0302cac75 - Update Citizen (#147)
[09:08:35] <RhinosF1>	 Reception123: deploying, jbr1,test2,mw[4-6]
[09:11:50] <RhinosF1>	 !log rebuildlc on jbr1,test1,mw[4-6]
[09:11:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[09:13:32] <Not-f378>	 [02miraheze/mediawiki] 07RhinosF1 deleted branch 03update-extskin
[09:13:33] <Not-f378>	 [02mediawiki] 07RhinosF1 deleted branch 03update-extskin - 13https://git.io/vbL5b
[09:14:13] <RhinosF1>	 live on test2 but puppet not finished on rest
[09:14:52] <RhinosF1>	 and puppet fails
[09:16:51] * RhinosF1 trying again
[09:18:22] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[09:18:27] <RhinosF1>	 ^ me
[09:18:31] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[09:18:51] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[09:19:02] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[09:20:18] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[09:20:28] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[09:20:30] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[09:20:52] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:21:02] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:21:05] <RhinosF1>	 ^ recovering, rebuilding i18n now
[09:22:34] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[09:28:32] * ZppixBot jealous, where's my updates?
[09:43:48] <Not-f378>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfh6c
[09:43:50] <Not-f378>	 [02miraheze/mw-config] 07Reception123 0386a8f27 - rm blacklist sitenotice
[09:49:18] <Reception123>	 RhinosF1: can we take a look at https://phabricator.miraheze.org/T5799 ?
[09:49:18] <ZppixBot>	 [ ⚓ T5799 Add Flow Discussion on Blog Page ] - phabricator.miraheze.org
[09:49:35] <Reception123>	 now that we've got you as a mw-admin I'd like to try to get more mediawiki tasks and feature requests resolved
[09:50:50] <RhinosF1>	 Reception123: I assume they mean https://www.mediawiki.org/wiki/Extension:BlogPage
[09:50:51] <ZppixBot>	 [ Extension:BlogPage - MediaWiki ] - www.mediawiki.org
[09:51:03] <Reception123>	 RhinosF1: yeah think so
[09:51:38] <RhinosF1>	 Reception123: Upstream then.
[09:57:24] <Reception123>	 RhinosF1: I'll ask paladox, maybe it can be enabled somewhere 
[09:57:27] <Reception123>	 if not we upstream and decline
[09:57:59] <RhinosF1>	 Reception123: it can't
[11:53:40] <avengium>	 hi
[11:54:10] <avengium>	 i want to have 3 edits in: https://tuscriaturas.miraheze.org/wiki/Especial:ManageWiki/settings#mw-section-restricted . ."editing attribution" up to 3 editors.
[11:54:13] <ZppixBot>	 [ Administra los ajustes adicionales de este wiki - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org
[11:54:45] <avengium>	 RhinosF1 is ok to ask it here?
[12:00:24] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfhD0
[12:00:26] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 03b10f29f - BOT: Updating services config for wikis
[12:02:08] <RhinosF1>	 avengium: yeah
[12:02:29] <avengium>	 ok
[12:14:23] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Static on bacula2 is WARNING: WARNING: Full, 6831642 files, 655.2GB, 2020-06-10 12:11:00 (2.1 weeks ago)
[12:24:11] <avengium>	 i need to do something for that to take effect?
[13:20:27] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfhH6
[13:20:29] <Not-f378>	 [02miraheze/puppet] 07paladox 03c10c33e - cloud[12]: Disable root login
[13:20:30] <Not-f378>	 [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS
[13:20:32] <Not-f378>	 [02puppet] 07paladox opened pull request 03#1413: cloud[12]: Disable root login - 13https://git.io/JfhHi
[13:20:49] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfhHX
[13:20:50] <Not-f378>	 [02miraheze/puppet] 07paladox 033080e63 - Update cloud2.yaml
[13:20:52] <Not-f378>	 [02puppet] 07paladox synchronize pull request 03#1413: cloud[12]: Disable root login - 13https://git.io/JfhHi
[13:20:58] <Not-f378>	 [02puppet] 07paladox closed pull request 03#1413: cloud[12]: Disable root login - 13https://git.io/JfhHi
[13:21:00] <Not-f378>	 [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/JfhH1
[13:21:01] <Not-f378>	 [02miraheze/puppet] 07paladox 0388a85b2 - Merge pull request #1413 from miraheze/paladox-patch-6 cloud[12]: Disable root login
[13:21:03] <Not-f378>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6
[13:21:04] <Not-f378>	 [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS
[14:27:49] <Not-f378>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfhdt
[14:27:51] <Not-f378>	 [02miraheze/ssl] 07paladox 033d48245 - Update miraheze.gq.crt
[14:28:18] <Not-f378>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfhdY
[14:28:19] <Not-f378>	 [02miraheze/ssl] 07paladox 038b988a8 - Update miraheze.ga.crt
[14:28:43] <Not-f378>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfhds
[14:28:44] <Not-f378>	 [02miraheze/ssl] 07paladox 038a75b6e - Update miraheze.ml.crt
[14:29:08] <Not-f378>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfhdn
[14:29:10] <Not-f378>	 [02miraheze/ssl] 07paladox 03fc4ef74 - Update miraheze.tk.crt
[14:30:22] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfhd4
[14:30:24] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 0331decb7 - BOT: Updating services config for wikis
[14:32:45] <icinga-miraheze>	 RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Wed 23 Sep 2020 13:27:27 GMT +0000.
[14:33:47] <icinga-miraheze>	 RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Wed 23 Sep 2020 13:28:29 GMT +0000.
[14:34:03] <icinga-miraheze>	 RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Wed 23 Sep 2020 13:28:01 GMT +0000.
[14:43:47] <icinga-miraheze>	 RECOVERY - miraheze.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.tk' will expire on Wed 23 Sep 2020 13:28:52 GMT +0000.
[14:47:16] <Left>	 Hey everytime i try and write a redirect page the name changes to the page it redirects to, any way i can disable this?
[14:48:07] <RhinosF1>	 Left: append ?redirect=no to the url or use the back button/link on the notice that says you've been redirected
[14:48:52] <RhinosF1>	 So if it was yourwiki.miraheze.org/wiki/Redirect you would do yourwiki.miraheze.org/Redirect?redirect=no
[15:12:34] <Left>	 ah ok
[15:13:05] <RhinosF1>	 Left: I mean yourwiki.miraheze.org/wiki/Redirect?redirect=no
[15:13:07] <RhinosF1>	 Sorry
[15:13:47] <Left>	 any way i can make it default
[15:14:42] <RhinosF1>	 Left: im not sure about that option
[15:32:49] <MH-Discord>	 <MrJaroslavik> Hey, is any template expert around?
[15:35:08] <MH-Discord>	 <⚠ I like bad words ⚠> what you need done?
[15:35:59] <MH-Discord>	 <MrJaroslavik> I need help with "if" parameter in the [[Meta:Wiki creators/List/row]] template. If is parameter {{{name2}}} added, it will be appear, like "</br>[[User:{{{name}}} |{{{name2}}}]]" if not, nothing appear.
[15:36:30] <MH-Discord>	 <⚠ I like bad words ⚠> thats prob a @RhinosF1  thing
[15:36:45] <MH-Discord>	 <MrJaroslavik> Sorry for the bad explanation too :D
[15:36:50] <MH-Discord>	 <⚠ I like bad words ⚠> its good
[15:37:22] <MH-Discord>	 <MrJaroslavik> Due to The_Pioneer, etc...
[15:37:25] <RhinosF1>	 I think {{{name2| {{{name}}} }}} might work
[15:41:04] <MH-Discord>	 <MrJaroslavik> I want use anything like here https://i.nuuls.com/qBY62.png
[15:41:44] <RhinosF1>	 Ah
[15:41:45] <MH-Discord>	 <MrJaroslavik> Or i can use normal template parameters instead of row template, for this row
[16:06:23] <icinga-miraheze>	 PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Sat 11 Jul 2020 16:03:46 GMT +0000).
[16:13:41] <Not-f378>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfhxL
[16:13:43] <Not-f378>	 [02miraheze/ssl] 07MirahezeSSLBot 034a7cdf4 - Bot: Update SSL cert for fc.songcontests.eu
[16:22:22] <icinga-miraheze>	 RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Wed 23 Sep 2020 15:13:35 GMT +0000.
[16:33:36] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 3.72, 2.75, 1.54
[16:39:30] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.93, 1.69, 1.44
[16:45:14] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfhpC
[16:45:15] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 0358ba578 - BOT: Updating services config for wikis
[16:46:25] <Not-f378>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfhpB
[16:46:26] <Not-f378>	 [02miraheze/mediawiki] 07paladox 03a8ba4cc - Update CheckUser
[17:00:05] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:00:32] <RhinosF1>	 JohnLewis: can you look at the task for ^ as well
[17:01:01] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:01:13] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:01:40] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:02:58] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures
[17:03:12] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:03:41] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:04:05] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:17:34] <JohnLewis>	 I looked and I’m unsure why I need to?
[17:17:35] <ZppixBot>	 JohnLewis: 2020-06-24 - 08:35:24UTC <RhinosF1> ask JohnLewis do you still need  https://github.com/miraheze/mw-config/tree/revert-3102-patch-2 as a branch?
[17:17:47] <JohnLewis>	 ive explained they reasons behind it about 3 tines now to Paladox 
[17:19:25] <RhinosF1>	 JohnLewis: can you explain it to me or propose a solution?
[17:20:10] <JohnLewis>	 If it’s increased anymore, it’s introduces the issue that it can run across multiple puppet runs meaning we don’t get deploys every 10 minutes
[17:20:59] <RhinosF1>	 Ah
[17:21:11] <RhinosF1>	 JohnLewis: what do we do about it failing first time then?
[17:21:48] <JohnLewis>	 Git clone toke needs to be massively reduced 
[17:21:56] <JohnLewis>	 *time
[17:22:21] <RhinosF1>	 JohnLewis: how do we do that?
[17:22:42] <JohnLewis>	 Smaller repo, faster network, faster write speeds to name a few
[17:23:33] <RhinosF1>	 JohnLewis: it couldnt be related to https://phabricator.miraheze.org/T5802 could it
[17:23:34] <ZppixBot>	 [ ⚓ T5802 CURL intermittently slow on mon1 ] - phabricator.miraheze.org
[17:23:59] <JohnLewis>	 No, that’s been happening for 5 years
[17:24:33] <RhinosF1>	 JohnLewis: i've never seen it as bad as it is now
[17:24:46] <Reception123>	 paladox also said he'd try to do a script so that services is updated
[17:24:54] <Reception123>	 that's the main reason we still have 10 min puppet runs
[17:25:02] <Reception123>	 or else we could make them like 30 mins / 1 hour
[17:25:16] <Reception123>	 (for the 10 min deploy explanation)
[17:25:31] <RhinosF1>	 So we move to SSDs as paladox always mentions them, not a clue how to speed the net up and smaller repo not happening
[17:26:18] <RhinosF1>	 Reception123: I wonder if we could have something check each repo if it has changed and then only pull then
[17:26:39] <JohnLewis>	 SSDs are expensive, that’s why we moved back to HDD. We had to make a decision and unfortunately super fast SSDs was the sacrifice 
[17:26:46] <Reception123>	 RhinosF1: well first we need a script to check changes and push services
[17:26:51] <Reception123>	 if we get that we can say goodbye to 10 min puppet runs 
[17:27:09] <RhinosF1>	 Or have something recieveing a message from github when repo's like notifico does and then schedule a puppet run
[17:27:26] <RhinosF1>	 JohnLewis: so it's making puppet more intelligent or speeding the net up
[17:27:39] <Reception123>	 RhinosF1: well what if different wikis enabled VE all in 5 mins that would mean 5 puppet runs
[17:27:54] <Reception123>	 we'd need something separate (paladox mentioned a python script) to do services 
[17:27:56] <RhinosF1>	 Reception123: rate limit it
[17:28:01] <RhinosF1>	 Or seperate it
[17:28:09] <Reception123>	 and then puppet runs kept at a minimum 
[17:28:41] <RhinosF1>	 Yeah
[17:29:26] <JohnLewis>	 The net should already be fairly good for our needs
[17:29:30] <RhinosF1>	 If we could move services to another script and then set a webhook up to record when the last change was on github and only run puppet if something has changed then it might work
[17:29:41] <RhinosF1>	 JohnLewis, paladox: does that sound possible?
[17:30:58] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.82, 1.88, 1.38
[17:32:37] <JohnLewis>	 GitHub has webhooks
[17:32:52] <RhinosF1>	 JohnLewis: it certainly does
[17:32:58] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.03, 1.60, 1.33
[17:34:17] <RhinosF1>	 As long as we can tweak puppet to save the webhook data and probably make it so the risk of someone sending a fake webhook to the server is reduced then it sounds possible
[17:34:32] <MH-Discord>	 <Void> I thought the issue was with the initial git pull, not any subsequent updates?
[17:34:43] <MH-Discord>	 <Void> *initial git clone
[17:35:18] <RhinosF1>	 @Void: no git pull, so I'm working out how  to not make it git pull things it doesn't need to
[17:35:37] <MH-Discord>	 <Void> Well, it doesn't
[17:36:27] <MH-Discord>	 <Void> It does a git fetch to determine if there have been changes, and if there have, then it git pulls
[17:36:39] <RhinosF1>	 Hmmm
[17:36:52] <RhinosF1>	 It'd nullify JohnLewis's concern about raising the timeout
[17:36:57] <MH-Discord>	 <Void> Unless someone changed that behavior since I last read the code
[17:37:07] <RhinosF1>	 As there wouldn't always be 10 minutely deploys
[17:37:08] <MH-Discord>	 <Avengium (Ángel)> Void, if you are a steward, you can check wiki/Requests_for_adoption 😃 👍
[17:37:44] <MH-Discord>	 <Avengium (Ángel)> sorry, for entering into the conversation all of a sudden 😅
[17:37:53] <RhinosF1>	 don't worry
[17:38:01] <JohnLewis>	 I don't really have a concern
[17:38:21] <MH-Discord>	 <Void> @Avengium (Ángel) will do in roughly 3 hours or so (mobile at the moment)
[17:38:22] <JohnLewis>	 It's a statement that, puppet runs every 10 minutes currently, therefore puppet can not run past 10 minutes
[17:38:46] <RhinosF1>	 JohnLewis: so if we make puppet not run every 10 minutes we'd be good
[17:39:20] <JohnLewis>	 Well, perhaps
[17:39:22] <JohnLewis>	 But
[17:39:37] <JohnLewis>	 Then the question to be answered is, why does this only happen occasionally?
[17:39:52] <JohnLewis>	 If it was to always happen - we'd never run updated code as puppet would always fail
[17:40:50] <RhinosF1>	 It's only the first puppet run after a mediawiki deploy
[17:41:07] <Reception123>	 yes it's only been when there's a mediawiki repo (not mw-config) change
[17:41:08] <RhinosF1>	 So something happens on the 1st that doesnt on subsequent
[17:41:17] <RhinosF1>	 Reception123: mediawiki is much bigger
[17:41:48] <Reception123>	 yeah
[17:42:15] <MH-Discord>	 <Void> Could just be that the whole thing hits the puppet timeout due to how long MediaWiki install can take
[17:42:36] * RhinosF1 pasted the puppet log on phab
[17:42:54] <MH-Discord>	 <Void> So it might not be that specific git clone that's the problem, but another step in the process
[17:43:58] <RhinosF1>	 Notice: /Stage[main]/Php::Php_fpm/Php::Extension[apcu]/Package[php7.3-apcu]/ensure: created (corrective)
[17:43:58] <RhinosF1>	 Error: Command exceeded timeout
[17:43:58] <RhinosF1>	 Error: /Stage[main]/Mediawiki/Git::Clone[MediaWiki core]/Exec[git_pull_MediaWiki core]/returns: change from 'notrun' to ['0'] failed: Command exceeded timeout (corrective)
[17:44:12] <Reception123>	 yeah that's what I've always seen too
[17:44:50] <MH-Discord>	 <Void> Is that the entire log, or just the error and the last successful step?
[17:45:16] <RhinosF1>	 @Void: https://phabricator.miraheze.org/T5803
[17:45:17] <ZppixBot>	 [ ⚓ T5803 Puppet failing after mediawiki deploys ] - phabricator.miraheze.org
[17:45:56] * RhinosF1 doesn't know where would have the logs from before that in the stels
[17:45:58] <RhinosF1>	 Steps
[17:49:01] <MH-Discord>	 <Void> Hmm, well I want to go through the puppet code anyway, so that might be a project for this weekend
[17:49:02] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.58, 2.70, 1.78
[17:52:59] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.94, 1.73, 1.59
[17:54:58] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.56, 1.35, 1.46
[19:43:38] <Naleksuh>	 i thought it was already banned
[20:21:53] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:23:51] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Thu 20 Aug 2020 14:59:34 GMT +0000.
[20:28:24] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2115 MB (10% inode=93%);
[20:45:13] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjIn
[20:45:15] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 038cd2325 - BOT: Updating services config for wikis
[21:18:32] <RhinosF1>	 Naleksuh: it was
[21:25:54] <Naleksuh>	 rhinosf1 : then why it was unbanned?
[21:26:49] <RhinosF1>	 Naleksuh: it wasnt, ban didnt catch it for some reasom
[21:27:29] <Voidwalker>	 It's no longer logged in
[21:35:14] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfjtC
[21:35:15] <Not-f378>	 [02miraheze/services] 07MirahezeSSLBot 03a18e6b4 - BOT: Updating services config for wikis
[23:15:03] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.87, 1.86, 1.20
[23:17:00] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.24, 1.54, 1.15
[23:29:01] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.56, 2.27, 1.64
[23:30:58] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.81, 1.74, 1.51
[23:32:57] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.87, 1.40, 1.41
[23:36:12] <icinga-miraheze>	 PROBLEM - misc1 Current Load on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:36:23] <icinga-miraheze>	 PROBLEM - misc1 Disk Space on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:36:54] <icinga-miraheze>	 PROBLEM - misc1 APT on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:37:15] <icinga-miraheze>	 PROBLEM - misc1 IMAP on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:38:00] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:40:14] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures
[23:40:52] <icinga-miraheze>	 RECOVERY - misc1 Disk Space on misc1 is OK: DISK OK - free space: / 34654 MB (85% inode=99%);
[23:41:18] <icinga-miraheze>	 RECOVERY - misc1 APT on misc1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[23:41:36] <icinga-miraheze>	 RECOVERY - misc1 IMAP on misc1 is OK: IMAP OK - 0.041 second response time on 185.52.1.76 port 143 [* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.]
[23:58:20] <icinga-miraheze>	 PROBLEM - misc1 Current Load on misc1 is WARNING: WARNING - load average: 0.00, 0.03, 1.83