[00:08:20] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 4137 files, 8.488MB, 2019-08-11 00:05:00 (1.1 weeks ago)
[02:36:35] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%);
[02:49:41] <icinga-miraheze>	 PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Full, 764611 files, 70.58GB, 2019-08-04 02:46:00 (2.1 weeks ago)
[06:26:35] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3273 MB (13% inode=94%);
[11:03:31] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.03, 1.58, 1.18
[11:05:30] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.77, 1.24, 1.10
[11:39:18] <Reception123>	 !m JohnLewis when you're around please take a look at why db5 doesn't want to work with wikis (connection error)
[11:39:18] <wm-bot5>	 You're doing good work, JohnLewis!
[11:39:26] <Reception123>	 .say JohnLewis when you're around please take a look at why db5 doesn't want to work with wikis (connection error)
[11:39:31] <Reception123>	 @tell JohnLewis when you're around please take a look at why db5 doesn't want to work with wikis (connection error)
[11:39:37] <Reception123>	 lol I'm so bad :P
[11:39:51] <wm-bot5>	 I will let you know when I see JohnLewis and I will deliver that message to them
[11:39:51] <Reception123>	 @notify JohnLewis when you're around please take a look at why db5 doesn't want to work with wikis (connection error)
[11:50:29] <icinga-miraheze>	 PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Wed 04 Sep 2019 11:47:15 AM GMT +0000).
[11:50:42] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFKO
[11:50:43] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot 03453dc96 - Bot: Update SSL cert for vmcodex.net
[11:52:38] <icinga-miraheze>	 PROBLEM - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bh.gyaanipedia.co.in' expires in 15 day(s) (Wed 04 Sep 2019 11:49:57 AM GMT +0000).
[11:52:51] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFKC
[11:52:52] <Not-f817>	 [02miraheze/ssl] 07MirahezeSSLBot 034a69808 - Bot: Update SSL cert for bh.gyaanipedia.co.in
[11:54:28] <icinga-miraheze>	 RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Sun 17 Nov 2019 10:50:35 AM GMT +0000.
[12:04:38] <icinga-miraheze>	 RECOVERY - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'bh.gyaanipedia.co.in' will expire on Sun 17 Nov 2019 10:52:44 AM GMT +0000.
[12:50:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjF6B
[12:50:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 031dee380 - BOT: Updating services config for wikis
[13:21:25] <Reception123>	 !log purge binary logs before '2019-08-19 06:00:00';
[13:21:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:10:13] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFP3
[14:10:15] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 03a6ae701 - BOT: Updating services config for wikis
[15:41:32] <Reception123>	 Voidwalker: hi. around?
[15:41:42] <Voidwalker>	 what's up?
[15:43:46] <Voidwalker>	 Reception123, ^
[15:52:46] <Reception123>	 Voidwalker: a local account attach
[15:52:49] <Reception123>	 will do the centralauth part now
[15:53:43] <RhinosF1>	 JohnLewis: hi, one of your renames is stuck on a deleted wiki
[15:54:14] <JohnLewis>	 RhinosF1: literally not blocked on me
[15:54:20] <JohnLewis>	 any sysadmin can resolve it
[15:54:30] <Reception123>	 JohnLewis: I've told you the issue in -staff
[15:54:46] <RhinosF1>	 JohnLewis: yeah speak to Reception123
[15:54:55] <JohnLewis>	 Reception123: I know, but easy fix is add to all.dblist quickly, then do the necessary stuff
[15:55:18] <Reception123>	 okay
[15:56:04] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$  sudo -u www-data php createLocalAccount.php --wiki=typhoonwikiwiki 'OwenFung87 '
[15:56:05] <Reception123>	 Voidwalker: ^
[15:56:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:56:21] <Reception123>	 JohnLewis: though the problem would always arise if a user is renamed while having an account on a 'deleted' wiki
[15:56:32] <JohnLewis>	 Reception123: Indeed
[15:56:43] <Reception123>	 and then there's also the deletedWikis.php bug
[15:57:06] <JohnLewis>	 I know, don't give me loads of stuff then expect them done immediately :)
[16:00:10] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjF1N
[16:00:12] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 0364f2524 - BOT: Updating services config for wikis
[16:03:30] <Reception123>	 !log temporarily re-added hypotheticalmetawiki to undeleted to resolve rename error
[16:03:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:03:37] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --logwiki metawiki --wiki metawiki Znotch190711 OwenFung87
[16:03:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:03:57] <Reception123>	 RhinosF1: fixed
[16:04:18] <RhinosF1>	 Reception123: won't Mirahezefakenewswiki cause the same issue
[16:04:22] <Reception123>	 meh, looks like there's more deleted wikis
[16:04:23] <Reception123>	 yeah just saw
[16:04:51] <RhinosF1>	 Yeah fix them all pls temporarily
[16:04:54] <Not-f817>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMT
[16:04:55] <Not-f817>	 [02miraheze/CreateWiki] 07JohnFLewis 036ad80b4 - continue execution, not return execution
[16:04:58] <Reception123>	 !log temporarily re-added mirahezefakenewswiki to undeleted to resolve rename error
[16:05:05] <JohnLewis>	 Reception123: ^ deleteWikis fixed
[16:05:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:05:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMk
[16:05:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 0363854ab - BOT: Updating services config for wikis
[16:05:15] <Reception123>	 JohnLewis: thanks :)
[16:06:09] <Reception123>	 !log re-ran script above to force account rename
[16:06:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:07:08] <RhinosF1>	 Reception123: you can remove from all.dblist now it's gone through on fake news
[16:07:25] <RhinosF1>	 And fully done now
[16:07:39] <JohnLewis>	 it'll be removed in a minute anyway
[16:07:54] <Reception123>	 :)
[16:10:09] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMO
[16:10:11] <Not-f817>	 [02miraheze/services] 07MirahezeSSLBot 033c46eb5 - BOT: Updating services config for wikis
[16:12:46] <JohnLewis>	 !log ran mediawiki grants for db5, wasn't done on install
[16:12:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:19:30] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMR
[16:19:32] <Not-f817>	 [02miraheze/mw-config] 07Reception123 035f1f89f - Revert "Revert "Revert "rm testwiki readonly for now""" This reverts commit 26b88da6a679497e2749f9f08dc54c75c650bfc2.
[16:27:02] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMa
[16:27:03] <Not-f817>	 [02miraheze/mw-config] 07Reception123 03a8e4965 - Revert "Revert "enable db5 wikis and add testwiki to db5 (#2732)"" This reverts commit c8d2e1c060be2d373aa16587a7db327e3ae5bc4f.
[16:36:21] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[16:38:54] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMS
[16:38:55] <Not-f817>	 [02miraheze/mw-config] 07Reception123 037baced8 - rm readonly for testwiki
[16:40:14] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:41:20] <Reception123>	 !log moved testwiki from db4 to db5 and DROP DATABASE testwiki on db4
[16:41:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:47:42] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.87, 6.16, 5.26
[16:48:43] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFMF
[16:48:45] <Not-f817>	 [02miraheze/mw-config] 07Reception123 034a11b49 - frikipediawiki sitenotice for db5 move
[16:49:40] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.95, 5.95, 5.28
[17:01:49] <RhinosF1>	 Reception123: still one last instruction on https://phabricator.miraheze.org/T4651
[17:01:50] <ZppixBot>	 [ ⚓ T4651 Import request for OsaIndex ] - phabricator.miraheze.org
[17:04:00] <Reception123>	 ah true
[17:12:04] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFDz
[17:12:06] <Not-f817>	 [02miraheze/mw-config] 07Reception123 03af60304 - frikipediawiki readonly for db5 migration
[17:26:05] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFD9
[17:26:07] <Not-f817>	 [02miraheze/mw-config] 07Reception123 038250a99 - add frikipediawiki to db5
[17:28:24] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFDb
[17:28:26] <Not-f817>	 [02miraheze/mw-config] 07Reception123 031a197aa - rm readonly on frikipediawiki
[17:30:29] <Not-f817>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjFDh
[17:30:31] <Not-f817>	 [02miraheze/mw-config] 07Reception123 035c6c9c7 - rm sitenotice for frikipediawiki
[17:30:39] <Reception123>	 !log moved frikipediawiki from db4 to db5 and DROP DATABASE frikipediawiki on db4
[17:30:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:12:12] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 41%
[18:14:07] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3%
[18:15:03] <paladox>	 Hmm
[18:51:44] <RhinosF1>	 paladox: aren't u on holiday?
[19:03:32] <paladox>	 Yeh
[19:10:34] <RhinosF1>	 paladox: then stop hmming and enjoys it
[19:10:50] * RhinosF1 says as he's supposed to be on sick leave while Wednesday
[19:31:16] <Reception123>	 lol both of you can't resist coming back :P
[19:31:31] <Reception123>	 though it's good that paladox is here to keep an eye on things, I wouldn't want to know he's completely gone for 2 weeks
[19:35:19] <paladox>	 Yup
[19:35:55] <RhinosF1>	 Reception123: true, and it's irresistible
[19:36:21] * RhinosF1 gets emailed that often for phab tasks he might as well check them
[19:58:33] * JohnLewis never gets Phab emails
[19:59:49] <paladox>	 JohnLewis: how? :P
[19:59:51] <paladox>	 You selected the “do  not send email” option?
[19:59:56] <JohnLewis>	 yep
[20:00:01] <paladox>	 Heh
[20:00:43] * RhinosF1 is sent
[20:00:55] <RhinosF1>	 Emails for every phab task he has access to
[20:02:22] <RhinosF1>	 And repo R9 changes (mw-config)
[20:02:48] <Hispano76>	 hii
[20:04:07] <RhinosF1>	 Hispano76: hi, what's up?
[21:09:09] <icinga-miraheze>	 PROBLEM - lizardfs3 Disk Space on lizardfs3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:11:08] <icinga-miraheze>	 PROBLEM - lizardfs3 Disk Space on lizardfs3 is WARNING: DISK WARNING - free space: / 11243 MB (7% inode=98%);
[21:13:01] <RhinosF1>	 JohnLewis: keep an eye out pls (icinga still indicates we've got a bit left though) ^
[21:16:36] <JohnLewis>	 RhinosF1: will do but the burn rate of static is so small
[21:17:43] <RhinosF1>	 JohnLewis: k cool, should be okay for a while then, let's just hope MKVM comes soon to NL
[21:22:35] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%);
[22:44:17] <Hispano76>	 nothing, I just say hello RhinosF1 
[22:47:19] <RhinosF1>	 Hispano76: well nice to hear from u
[22:47:41] <Hispano76>	 :)
[22:47:57] <RhinosF1>	 :)