[01:05:11] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPZv
[01:05:13] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 031c4456c - BOT: Updating services config for wikis
[02:25:12] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPc7
[02:25:14] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 03c077d4c - BOT: Updating services config for wikis
[02:46:57] <icinga-miraheze>	 PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 59%
[03:05:06] <icinga-miraheze>	 RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 36%
[07:50:20] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfP0T
[07:50:21] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 0394ce66a - BOT: Updating services config for wikis
[10:53:56] <Not-1447>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPre
[10:53:58] <Not-1447>	 [02miraheze/mw-config] 07JohnFLewis 03b19543a - repository->repositories
[11:10:32] <Not-1447>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPro
[11:10:33] <Not-1447>	 [02miraheze/mw-config] 07JohnFLewis 03cbf28e2 - make prefixMapping an array of blank string keys
[11:14:36] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[11:14:56] <Not-1447>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPrS
[11:14:58] <Not-1447>	 [02miraheze/mw-config] 07JohnFLewis 03edd7874 - client sql needs to mirror repo for case of remote external clients
[12:29:35] <icinga-miraheze>	 PROBLEM - db6 Current Load on db6 is CRITICAL: CRITICAL - load average: 10.65, 7.99, 5.56
[12:30:45] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 34.13, 27.85, 20.07
[12:57:46] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 35.11, 18.94, 12.05
[12:57:58] <icinga-miraheze>	 PROBLEM - db6 Current Load on db6 is WARNING: WARNING - load average: 3.14, 6.23, 7.68
[12:59:46] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.66, 19.89, 13.34
[12:59:46] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 10.15, 16.73, 23.30
[13:03:53] <icinga-miraheze>	 RECOVERY - db6 Current Load on db6 is OK: OK - load average: 2.96, 4.15, 6.32
[13:06:04] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 8.96, 11.91, 19.06
[13:08:42] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.55, 3.26, 2.32
[13:11:25] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.85, 24.42, 18.73
[13:12:54] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.87, 3.48, 2.68
[13:13:37] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 16.32, 22.37, 18.80
[13:14:50] <icinga-miraheze>	 RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.76, 2.81, 2.53
[13:15:35] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.98, 18.48, 17.76
[13:23:57] <SPF|Cloud>	 !log disable puppet on test2 to test TLS chain change
[13:24:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:25:34] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 1 failures
[13:28:59] <Not-1447>	 [02puppet] 07Amanda-Catherine opened pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfPXL
[13:36:13] <AmandaCath>	 Good morning 
[13:36:18] <RhinosF1>	 hi
[13:36:24] <Reception123>	 hi
[13:37:05] <AmandaCath>	 I'd assume from the fact that almost everyone is currently +o that more spambots have been flooding the channel...
[13:47:56] <AmandaCath>	 .tell Examknow please see your Meta talk page
[13:47:59] <ZppixBot>	 AmandaCath: I'll pass that on when Examknow is around.
[13:48:58] <paladox>	 !log upgrade grafana on mon1
[13:49:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:49:31] <Sario>	 AmandaCath: Examknow has changed their irc nick to Texas
[13:49:50] <AmandaCath>	 Oh
[13:50:48] <SPF|Cloud>	 !log delete port 3307 rule on dbt1
[13:50:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:55:41] <RhinosF1>	 SPF|Cloud: For the twitter api,  We need a phone number, do you still have the miraheze one we could use too verify?
[14:15:20] <icinga-miraheze>	 PROBLEM - db7 APT on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:15:30] <icinga-miraheze>	 PROBLEM - db7 Puppet on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:15:39] <icinga-miraheze>	 PROBLEM - ping4 on db7 is CRITICAL: PING CRITICAL - Packet loss = 100%
[14:15:46] <icinga-miraheze>	 PROBLEM - db7 Disk Space on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:16:21] <AmandaCath>	 Uh oh
[14:16:22] <SPF|Cloud>	 !log reboot db7
[14:16:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:16:46] <icinga-miraheze>	 PROBLEM - Host db7 is DOWN: PING CRITICAL - Packet loss = 100%
[14:16:47] <SPF|Cloud>	 AmandaCath: it's the replica, no production impact :)
[14:17:02] <SPF|Cloud>	 but we broke the networking so had to give it a reboot
[14:17:27] <AmandaCath>	 Well, at least there's no production impact
[14:17:36] <AmandaCath>	 But breaking our own networking doesn't sound good
[14:20:12] <paladox>	 !log stopping mysql on db7 (service mysql stop)
[14:20:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:20:35] <paladox>	 well only ipv6 was broken
[14:21:10] <SPF|Cloud>	 paladox and I are testing a fix for the certificate issues we had a few days ago, but apparently ipv6 was broken on db7
[14:21:50] <AmandaCath>	 Oh, still debugging the recent site outage?
[14:22:50] <SPF|Cloud>	 the cause is known, but properly fixing things isn't as easy as we hoped
[14:23:48] <SPF|Cloud>	 although this downtime was mostly bad luck, a certificate that expired after twenty years :P
[14:24:08] <AmandaCath>	 Wow
[14:24:28] <paladox>	 !log reboot db7
[14:24:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:24:43] <icinga-miraheze>	 RECOVERY - Host db7 is UP: PING OK - Packet loss = 79%, RTA = 0.22 ms
[14:24:45] <icinga-miraheze>	 PROBLEM - ping6 on db7 is CRITICAL: CRITICAL - Destination Unreachable (fe80::ff:fe0b:abc)
[14:24:45] <icinga-miraheze>	 PROBLEM - db7 Current Load on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:24:45] <icinga-miraheze>	 PROBLEM - db7 SSH on db7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:24:45] <icinga-miraheze>	 PROBLEM - db7 MySQL on db7 is CRITICAL: Can't connect to MySQL server on '51.89.160.143' (115)
[14:24:47] <icinga-miraheze>	 PROBLEM - db7 SSH on db7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:24:47] <icinga-miraheze>	 PROBLEM - db7 Current Load on db7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[14:24:47] <icinga-miraheze>	 PROBLEM - ping6 on db7 is CRITICAL: CRITICAL - Destination Unreachable (fe80::ff:fe0b:abc)
[14:24:47] <icinga-miraheze>	 PROBLEM - db7 MySQL on db7 is CRITICAL: Can't connect to MySQL server on '51.89.160.143' (115)
[14:25:13] <icinga-miraheze>	 RECOVERY - db7 APT on db7 is OK: APT OK: 29 packages available for upgrade (0 critical updates).
[14:25:20] <icinga-miraheze>	 RECOVERY - db7 Disk Space on db7 is OK: DISK OK - free space: / 161682 MB (48% inode=99%);
[14:25:27] <icinga-miraheze>	 RECOVERY - ping4 on db7 is OK: PING OK - Packet loss = 0%, RTA = 0.44 ms
[14:25:37] <SPF|Cloud>	 yep, we have ipv6 now :)
[14:26:01] <paladox>	 yup!
[14:26:08] <icinga-miraheze>	 RECOVERY - ping6 on db7 is OK: PING OK - Packet loss = 0%, RTA = 0.27 ms
[14:26:16] <paladox>	 mysql starting up
[14:26:21] <icinga-miraheze>	 RECOVERY - db7 SSH on db7 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0)
[14:26:27] <icinga-miraheze>	 RECOVERY - db7 Current Load on db7 is OK: OK - load average: 0.99, 0.50, 0.19
[14:26:39] <icinga-miraheze>	 RECOVERY - db7 MySQL on db7 is OK: Uptime: 118 Threads: 8 Questions: 1031 Slow queries: 42 Opens: 157 Flush tables: 1 Open tables: 151 Queries per second avg: 8.737
[14:27:10] <icinga-miraheze>	 RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:28:49] <Sario>	 The icinga Litany of Recovery is always nice to see
[14:29:04] <MH-Discord>	 <xxEzraBxxx> people seem to cant login on my wiki
[14:29:13] <MH-Discord>	 <xxEzraBxxx> due to a hijacking and submit a form thing
[14:29:19] <MH-Discord>	 <xxEzraBxxx> do they need to clear their cookies?
[14:29:31] <Sario>	 What wiki?
[14:29:47] <MH-Discord>	 <xxEzraBxxx> 8b8tofficial.miraheze.org
[14:29:53] <MH-Discord>	 <xxEzraBxxx> for a minecraft anarchy server
[14:30:03] <MH-Discord>	 <xxEzraBxxx> People can create new accounts
[14:30:13] <MH-Discord>	 <xxEzraBxxx> But before a certain date existing ones couldnt log in
[14:30:25] <MH-Discord>	 <xxEzraBxxx> and I am instructing them to clear their cookies
[14:30:33] <MH-Discord>	 <xxEzraBxxx> Because it worked for me
[14:31:15] <AmandaCath>	 The "session hijacking" thing is a long-standing error
[14:31:36] <AmandaCath>	 Usually just reentering the password WITHOUT reloading the page or anything else will do the trick
[14:34:07] <MH-Discord>	 <xxEzraBxxx> i instructed users to clear their cookies just bow
[14:34:15] <MH-Discord>	 <xxEzraBxxx> because my other admins cant log in
[14:34:42] <AmandaCath>	 Clearing cookies will sometimes work, though usually it's not necessary
[14:35:02] <Not-1447>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPDl
[14:35:04] <Not-1447>	 [02miraheze/puppet] 07Southparkfan 037edeb8b - Set ssl=0 in root my.cnf
[14:35:32] <MH-Discord>	 <xxEzraBxxx> Amanda just safe precaution
[14:35:50] <MH-Discord>	 <xxEzraBxxx> once I did that never had any logging in issues for my account
[14:40:19] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPDr
[14:40:21] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 037930cd7 - BOT: Updating services config for wikis
[14:51:12] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfPyT
[14:51:14] <Not-1447>	 [02miraheze/mw-config] 07paladox 034b96676 - Database: Change sslCAFile back to /etc/ssl/certs/Sectigo.crt
[14:51:15] <Not-1447>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[14:51:17] <Not-1447>	 [02mw-config] 07paladox opened pull request 03#3095: Database: Change sslCAFile back to /etc/ssl/certs/Sectigo.crt - 13https://git.io/JfPyk
[14:53:12] <Not-1447>	 [02mw-config] 07paladox closed pull request 03#3095: Database: Change sslCAFile back to /etc/ssl/certs/Sectigo.crt - 13https://git.io/JfPyk
[14:53:13] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPyL
[14:53:15] <Not-1447>	 [02miraheze/mw-config] 07paladox 03027a676 - Database: Change sslCAFile back to /etc/ssl/certs/Sectigo.crt (#3095)
[14:53:36] <icinga-miraheze>	 RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[14:53:48] <icinga-miraheze>	 RECOVERY - dbt1 Puppet on dbt1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:54:03] <travis-ci>	 miraheze/mw-config/paladox-patch-2/4b96676 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/694292447
[14:54:16] <Not-1447>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[14:54:17] <Not-1447>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[14:54:37] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfPyq
[14:54:39] <Not-1447>	 [02miraheze/mw-config] 07paladox 03bb1134b - Change to sslCAPath
[14:54:40] <Not-1447>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[14:54:42] <Not-1447>	 [02mw-config] 07paladox opened pull request 03#3096: Change to sslCAPath - 13https://git.io/JfPym
[14:54:59] <Not-1447>	 [02mw-config] 07paladox closed pull request 03#3096: Change to sslCAPath - 13https://git.io/JfPym
[14:55:00] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPyY
[14:55:02] <Not-1447>	 [02miraheze/mw-config] 07paladox 03e0c93cc - Change to sslCAPath (#3096)
[14:55:52] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4
[14:57:51] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy
[15:02:33] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[15:29:21] <Not-1447>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[15:29:23] <Not-1447>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[15:32:41] <AmandaCath>	 SRE, in case you didn't notice https://github.com/miraheze/puppet/pull/1388
[15:32:42] <ZppixBot>	 [ Add Reddit and Dropbox to CSP whitelist T5685 by Amanda-Catherine · Pull Request #1388 · miraheze/puppet · GitHub ] - github.com
[15:33:15] <AmandaCath>	 Also someone with more experience at complex configuration variables should look at T5687
[15:45:49] <paladox>	 !log MariaDB [test2wiki]> truncate objectcache; - db6
[15:45:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:48:10] <paladox>	 !log MariaDB [test2wiki]> truncate objectcache; - db7
[15:48:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:48:49] <Reception123>	 AmandaCath: since there isn't a clear policy as of yet, that would need SPF|Cloud to sign off 
[16:10:19] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPHN
[16:10:21] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 03585aabb - BOT: Updating services config for wikis
[16:11:16] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfPHx
[16:11:17] <Not-1447>	 [02miraheze/mw-config] 07paladox 031448c80 - Site notice for db maintenance
[16:11:19] <Not-1447>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[16:11:20] <Not-1447>	 [02mw-config] 07paladox opened pull request 03#3097: Site notice for db maintenance - 13https://git.io/JfPHp
[16:11:35] <hispano76>	 https://phabricator.miraheze.org/T5688 Any comments?
[16:11:36] <ZppixBot>	 [ ⚓ T5688 Importing Wikidata properties into Hispanowiki ] - phabricator.miraheze.org
[16:14:18] <Not-1447>	 [02mw-config] 07paladox synchronize pull request 03#3097: Site notice for db maintenance - 13https://git.io/JfPHp
[16:14:19] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfPQJ
[16:14:21] <Not-1447>	 [02miraheze/mw-config] 07paladox 03d73150c - Update Sitenotice.php
[16:14:56] <Not-1447>	 [02mw-config] 07paladox closed pull request 03#3097: Site notice for db maintenance - 13https://git.io/JfPHp
[16:14:57] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPQT
[16:14:59] <Not-1447>	 [02miraheze/mw-config] 07paladox 0328eb9f5 - Site notice for db maintenance (#3097)
[16:15:00] <Not-1447>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[16:15:02] <Not-1447>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[16:15:59] <travis-ci>	 miraheze/mw-config/paladox-patch-2/d73150c - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/694325002
[17:24:35] <Texas>	 aww wtf??
[17:29:45] <MH-Discord>	 <MrJaroslavik> Anyone here can add Gadget MediaWiki:Gadget-CleanDeleteReasons.js,please?
[17:32:49] <Reception123>	 @MrJaroslavik aren't you a temporary sysop now? :)
[17:33:20] <MH-Discord>	 <MrJaroslavik> I dont have perms for edit/create .JS pages ;)
[17:33:28] <Reception123>	 oh
[17:34:02] <Reception123>	 @MrJaroslavik where can I copy the js code from?
[17:34:03] <MH-Discord>	 <MrJaroslavik> You can do it?
[17:34:07] <MH-Discord>	 <MrJaroslavik> https://en.wikibooks.org/w/index.php?title=MediaWiki:Gadget-CleanDeleteReasons
[17:34:08] <ZppixBot>	 [ MediaWiki:Gadget-CleanDeleteReasons - Wikibooks, open books for an open world ] - en.wikibooks.org
[17:34:10] <MH-Discord>	 <MrJaroslavik> https://en.wikibooks.org/w/index.php?title=MediaWiki:Gadget-CleanDeleteReasons.js
[17:34:10] <ZppixBot>	 [ MediaWiki:Gadget-CleanDeleteReasons.js - Wikibooks, open books for an open world ] - en.wikibooks.org
[17:34:15] <MH-Discord>	 <MrJaroslavik> From here
[17:34:16] <Reception123>	 ok
[17:34:17] <MH-Discord>	 <MrJaroslavik> Thanks
[17:35:47] <Reception123>	 @MrJaroslavik done
[17:37:38] <MH-Discord>	 <MrJaroslavik> Thanks!
[17:38:20] <Reception123>	 you're welcome :D
[17:38:54] <MH-Discord>	 <MrJaroslavik> Thanks! <3 i will work on any templates today :D
[17:39:57] <Reception123>	 :)
[17:40:07] <ZppixBot>	 Hello gognurk! If you have any questions, feel free to ask and someone should answer soon.
[17:40:17] <gognurk>	 HEY PALADOX WILL YOU SING WITH ME?
[17:49:34] <gognurkBOT>	 hiii paladox!
[17:49:50] <gognurkBOT>	 will you SING WITH ME??????????????????????????
[17:49:57] <Texas>	 paladox: Can we get a perm ban on his host?
[17:50:09] <Texas>	 SigmaBot only bans for 5 mins in here
[17:50:19] <Texas>	 so eir will remove in a few mins
[17:51:33] <Texas>	 ty
[18:22:02] <paladox>	 !log rebuild lc on test2
[18:22:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:25:12] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPbv
[18:25:13] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 037fd5277 - BOT: Updating services config for wikis
[19:01:44] <paladox>	 !log restarting mysql on dbt1 & db6
[19:01:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:04:10] <icinga-miraheze>	 PROBLEM - dbt1 MySQL on dbt1 is CRITICAL: Can't connect to MySQL server on '51.77.109.151' (115)
[19:04:27] <icinga-miraheze>	 PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7
[19:04:29] <icinga-miraheze>	 PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 40%
[19:04:37] <RhinosF1>	 ty
[19:05:09] <MH-Discord>	 <MrJaroslavik> Backend fetch failed
[19:05:14] <MH-Discord>	 <MrJaroslavik> :(
[19:05:37] <MH-Discord>	 <dreamcast99> this was expected
[19:05:40] <RhinosF1>	 ^
[19:05:43] <MH-Discord>	 <dreamcast99> they are rebooting the db
[19:11:34] <ZppixBot>	 Hello aliai! If you have any questions, feel free to ask and someone should answer soon.
[19:16:52] <Not-1447>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JfPAv
[19:16:54] <Not-1447>	 [02miraheze/puppet] 07paladox 0355e16dc - Revert "grafana: Temporarily change server_cert_name to db6.miraheze.org" This reverts commit df17737cb0422538ef09e1cca5804b90c5d9428e.
[19:16:55] <Not-1447>	 [02miraheze/puppet] 07paladox 038d5ad0a - Revert "Grafana: use LE as CA for database connection" This reverts commit bc09cce6d618daeb5debda15e984e774015595a4.
[19:30:28] <Not-1447>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JfPA2
[19:30:30] <Not-1447>	 [02miraheze/puppet] 07paladox 03b72205f - Revert "Icinga: use LE as CA for database connection" This reverts commit eb04c8bac13d2a32cffeb0b29ef7aa5484a5dcfb.
[19:30:31] <Not-1447>	 [02miraheze/puppet] 07paladox 0329f484f - Revert "Matomo: use LE as CA for database connection" This reverts commit f34cd59909f7d7ae75d6b8ffb5fb961c2d90c91e.
[19:55:11] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPx0
[19:55:13] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 031f8b621 - BOT: Updating services config for wikis
[19:59:55] <Not-1447>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JfPx6
[19:59:56] <Not-1447>	 [02miraheze/ssl] 07paladox 034313db4 - Fix Sectigo certificate
[19:59:58] <Not-1447>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[20:00:02] <Not-1447>	 [02ssl] 07paladox opened pull request 03#316: Fix Sectigo certificate - 13https://git.io/JfPxi
[20:00:24] <Not-1447>	 [02ssl] 07paladox closed pull request 03#316: Fix Sectigo certificate - 13https://git.io/JfPxi
[20:00:26] <Not-1447>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPxX
[20:00:27] <Not-1447>	 [02miraheze/ssl] 07paladox 03fd86988 - Fix Sectigo certificate (#316)
[20:00:29] <Not-1447>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[20:00:30] <Not-1447>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[20:05:12] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPxF
[20:05:13] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 03d2987f8 - BOT: Updating services config for wikis
[20:07:58] <paladox>	 !log reload nginx on mw*, jobrunner1 & cp*
[20:08:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:10:20] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPxj
[20:10:22] <Not-1447>	 [02miraheze/mw-config] 07paladox 035240025 - Revert "Change to sslCAPath (#3096)" This reverts commit e0c93ccb35bb6a53fadce0ca54397e7f2f648738.
[20:12:38] <paladox>	 !log MariaDB [(none)]> CHANGE MASTER TO MASTER_SSL_CA='/etc/ssl/certs/Sectigo.crt', MASTER_SSL_CERT='/etc/ssl/certs/wildcard.miraheze.org.crt', MASTER_SSL_KEY='/etc/ssl/private/wildcard.miraheze.org.key'; - db7
[20:12:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:14:26] <paladox>	 !log MariaDB [test2wiki]> truncate objectcache; - db6 & db7
[20:14:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:18:45] <RhinosF1>	 :)
[20:18:50] <RhinosF1>	 Well done everyone
[20:19:26] <paladox>	 thanks go to SPF|Cloud 
[20:20:18] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPpl
[20:20:20] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 03d6ff9d4 - BOT: Updating services config for wikis
[20:20:36] * RhinosF1 looks at SPF|Cloud and smiles
[20:20:37] <Sario>	 .gj SPF|Cloud 
[20:20:37] <ZppixBot>	 You're doing good work, SPF|Cloud !
[20:21:06] <Texas>	 way to go SPF|Cloud :P
[20:23:44] <SPF|Cloud>	 please credit paladox too, he did the actual deployment
[20:24:25] <SPF|Cloud>	 I found out how to fix it, but someone has to follow my plans... which takes time as well
[20:26:31] * RhinosF1 would say he owes you a drink but paladox doesnt drink
[20:26:41] <RhinosF1>	 Not sure about you
[20:31:21] <ZppixBot>	 Hello locococo2016! If you have any questions, feel free to ask and someone should answer soon.
[20:45:01] <hispano76>	 they removed the sitenotice?
[20:45:25] <MH-Discord>	 <MrJaroslavik> About IA group you mean?
[20:45:50] <paladox>	 done
[20:45:53] <Not-1447>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfPh0
[20:45:54] <Not-1447>	 [02miraheze/mw-config] 07paladox 036aca5e5 - Revert "Site notice for db maintenance (#3097)" This reverts commit 28eb9f5975c23e3a35daa6b884217c47257168a4.
[20:46:13] <paladox>	 @MrJaroslavik he means about the db maintenance
[20:46:16] <hispano76>	 ah, okay
[20:46:32] <MH-Discord>	 <MrJaroslavik> Ok, sorry
[22:05:11] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfXe6
[22:05:13] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 0393cbd12 - BOT: Updating services config for wikis