[00:03:03] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:07] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:13] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:15] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:39] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:45] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:04:53] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:11:07] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures  
[00:11:13] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures  
[00:11:15] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures  
[00:12:45] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:12:53] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:13:03] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:13:05] <icinga-miraheze>	 RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures  
[00:13:39] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:40:57] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures  
[05:00:20] <Not-2d3>	 [02puppet] 07JohnFLewis commented on pull request 03#897: Split up Volumes in bacula - 13https://git.io/fp8Er
[10:23:03] <icinga-miraheze>	 PROBLEM - kkutu.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kkutu.wiki' expires in 15 day(s) (Fri 07 Dec 2018 10:20:18 AM GMT +0000).  
[10:23:18] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp8yA
[10:23:19] <Not-2d3>	 [02miraheze/ssl] 07MirahezeSSLBot 0301a3c23 - Bot: Update SSL cert for kkutu.wiki
[10:31:04] <icinga-miraheze>	 RECOVERY - kkutu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kkutu.wiki' will expire on Tue 19 Feb 2019 09:23:12 AM GMT +0000.  
[12:01:05] <ZppixBot>	 Hello drqxr! If you have any questions feel free to ask and someone should answer soon.
[12:01:16] <drqxr>	 hi
[12:02:21] <drqxr>	 I am trying to create an account but I can't - getting a database query error.
[15:06:30] <MacFan4000>	 !log renaming anothertimeline2120wiki to megrezwiki
[15:06:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:19:19] <Reception123>	 MacFan4000: it's possible for mw-admins to rename now? 
[17:19:22] <Reception123>	 I wasn't aware...
[17:19:49] <MacFan4000>	 Yeah
[17:20:40] <Reception123>	 MacFan4000: really? How do you have access to renaming the database?
[17:20:45] <MacFan4000>	 mysql and mysqldump work on mw1 when I do -h db4.miraheze.org and use the credentials from PrivateSettings.php
[17:21:39] <MacFan4000>	 i create a dump then import the the new db from the dump
[17:23:56] <Reception123>	 MacFan4000: I'm not sure you should be doing that as mw-admin, however, without proper access to db4 and without being part of the Ops team
[17:27:42] <MacFan4000>	 Note that I don’t do any dropping
[17:27:49] <MacFan4000>	 I leave that part to ops
[17:29:33] <MacFan4000>	 It’s very similar process to https://meta.miraheze.org/wiki/Tech:Rename_a_wiki
[17:29:34] <ZppixBot>	 Title: [ Tech:Rename a wiki - Miraheze Meta ] - meta.miraheze.org
[17:30:41] <JohnLewis>	 “don’t do any dropping” but that’s part of the process and sometimes until you drop it, there’s no verbose issues
[17:33:11] <MacFan4000>	 You know that there aren’t issues when you successfully visit the wiki at the new URL
[17:33:30] <MacFan4000>	 Also theese commands only work on mw1
[17:33:50] <MacFan4000>	 They don’t work on any other mw server
[17:34:40] <MacFan4000>	 Dropping is also the very last step
[17:38:02] <JohnLewis>	 mysql* shouldn't be on any server though
[17:38:25] <MacFan4000>	 Sounds to me like somebody installed it on mw1
[17:38:45] <Reception123>	 paladox: ^ know anything about this?
[17:38:54] <paladox>	 Nope i do not
[17:39:15] <paladox>	 Anyways should i remove it JohnLewis ?
[17:39:21] <MacFan4000>	 I only discovered this today
[17:39:32] <JohnLewis>	 E: Unable to locate package mysqldump
[17:39:32] <JohnLewis>	 root@mw1:~# mysqldump
[17:39:32] <JohnLewis>	 Usage: mysqldump [OPTIONS] database [tables]
[17:39:36] <JohnLewis>	 heh apt
[17:40:03] <paladox>	 JohnLewis thats because it's provided by mysql-client
[17:40:04] <paladox>	 :P
[17:40:09] <paladox>	 there's no mysqldump package
[17:40:34] <JohnLewis>	 !log remove MariaDB* on mw1
[17:40:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:50:47] <MacFan4000>	 Something is still making those commands usable
[17:53:15] <MacFan4000>	 just did apt list —installed and I can see that it says mysql-common is installed
[17:54:11] <paladox>	 !log apt-get --purge remove mysql*
[17:54:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:00:38] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#897: Split up Volumes in bacula - 13https://git.io/fp8t9
[18:00:39] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fp40q
[18:00:41] <Not-2d3>	 [02miraheze/puppet] 07paladox 03aabf1c8 - Update bacula-dir.conf
[18:00:51] <Not-2d3>	 [02puppet] 07paladox edited pull request 03#897: Split up Volumes in bacula - 13https://git.io/fp8t9
[18:01:38] <JohnLewis>	 paladox: please update the data comment as well please
[18:02:06] <paladox>	 JohnLewis this one "# Pools - current maximum disk usage: 652G / 491G (!)"?
[18:02:12] <JohnLewis>	 yes
[18:02:38] <paladox>	 woulden't that make it
[18:02:53] <paladox>	 692gb?
[18:02:54] <paladox>	 JohnLewis ^^
[18:03:05] <JohnLewis>	 no, it's wrong already
[18:03:16] <JohnLewis>	 because you already made changes without updating it
[18:04:06] <paladox>	 um
[18:04:31] <paladox>	 what should it be?
[18:04:41] <JohnLewis>	 calculate it...
[18:06:08] <paladox>	 JohnLewis 662
[18:06:16] <paladox>	 240 + 400 + 20 + 2
[18:06:16] <JohnLewis>	 okay
[18:07:00] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#897: Split up Volumes in bacula - 13https://git.io/fp8t9
[18:07:02] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fp40b
[18:07:03] <Not-2d3>	 [02miraheze/puppet] 07paladox 03ee49a09 - Update bacula-dir.conf
[18:07:07] <paladox>	 JohnLewis ^
[18:07:25] <JohnLewis>	 k
[19:26:14] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp4az
[19:26:15] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 0324b75ad - Whitelist Special:Interwiki
[19:29:12] <Not-2d3>	 [02puppet] 07paladox closed pull request 03#897: Split up Volumes in bacula - 13https://git.io/fp8t9
[19:29:14] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp4a6
[19:29:15] <Not-2d3>	 [02miraheze/puppet] 07paladox 03f6fe153 - Split up Volumes in bacula (#897) * Split up Volumes in bacula This changes DB4 to 40gb x 5 = 200gb. This also changes static backup too. * Update bacula-dir.conf * Update bacula-dir.conf
[19:29:17] <Not-2d3>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[19:29:18] <Not-2d3>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[19:47:57] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp4wT
[19:47:58] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 03e1f0658 - Add some Lang specific entry’s
[19:50:40] <JohnLewis>	 should probably use a hook rather than do a long list
[19:53:10] <Not-2d3>	 [02mw-config] 07MacFan4000 deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[19:53:12] <Not-2d3>	 [02miraheze/mw-config] 07MacFan4000 deleted branch 03paladox-patch-1
[20:32:05] <paladox>	 !log sudo apt-get install python-certbot-nginx -t stretch-backports on mw1 for T3822
[20:32:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:42:34] <paladox>	 !log apt-get --purge remove python-certbot*
[20:42:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:42:57] <paladox>	 !log sudo apt-get install certbot -t stretch-backports on mw1
[20:43:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:57:25] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp46P
[20:57:26] <Not-2d3>	 [02miraheze/puppet] 07paladox 035aaeeb5 - Redirect /.well-known/acme-challenge to mw1
[21:14:10] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp4id
[21:14:12] <Not-2d3>	 [02miraheze/puppet] 07paladox 03ec9ecaa - Revert "Redirect /.well-known/acme-challenge to mw1" This reverts commit 5aaeeb534a16a09744a71923a7ce503576a5bf8c.
[21:19:03] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures  
[21:30:49] <MH-Discord>	 <subnekosis> does anyone know if im allowed to have more than 1 wiki per acount?
[21:31:33] <paladox>	 Yes you can. (we doin't have a policy that prevents you as far as i know)
[21:32:33] <MH-Discord>	 <subnekosis> ok thank you ^-^
[21:32:34] <MH-Discord>	 <subnekosis> also
[21:32:35] <MH-Discord>	 <subnekosis> umm
[21:32:37] <MH-Discord>	 <subnekosis> weird question
[21:32:45] <MH-Discord>	 <subnekosis> how many people see the "reason for wiki"?
[21:35:14] <MH-Discord>	 <subnekosis> actually nvm wont matter ^-^
[21:35:18] <MH-Discord>	 <subnekosis> i can be vague but truthful
[21:39:10] <Voidwalker>	 paladox, checked SN recently?
[21:39:32] <Voidwalker>	 looks like DB/Internal errors on user creations and wiki creations
[21:40:43] <paladox>	 uh
[21:40:47] * paladox looks
[21:44:37] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp41T
[21:44:38] <Not-2d3>	 [02miraheze/mw-config] 07paladox 037ae4a9f - Set wgCreateWikiCDBDirectory
[21:46:26] <MH-Discord>	 <subnekosis> miraheze has just crashed
[21:46:32] <MH-Discord>	 <subnekosis> where do i post crash reports?
[21:46:35] <MH-Discord>	 <subnekosis> and links
[21:46:54] <paladox>	 @subnekosis hi, known
[21:47:00] <paladox>	 JohnLewis CW is broken
[21:47:11] <paladox>	 see -staff
[21:47:12] <MH-Discord>	 <subnekosis> its back ^-^
[21:52:14] <Not-2d3>	 [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_31 [+0/-0/±2] 13https://git.io/fp41X
[21:52:15] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 03fb6493e - Revert "update CW" This reverts commit 5b8e6e9cf5f4af89595e782b2039031e410db45f.
[21:52:17] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 03bdbcb2d - Revert "update CW" This reverts commit 1722682ac530e8e4b2da538be8d09b5aa7a2ae59.
[21:55:43] <paladox>	 JohnLewis actually now it's causing wiki outages so i reverted your commits ^^
[21:55:49] <paladox>	 see https://phabricator.miraheze.org/T3823
[21:55:49] <ZppixBot>	 Title: [ ⚓ T3823 CreateWiki is broken @ master ] - phabricator.miraheze.org
[22:10:26] <paladox>	 !log recreating techbestpracticewiki
[22:10:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:16:45] <paladox>	 !log recreating suginamisannekiwiki
[22:16:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:18:19] <paladox>	 oh :(
[22:24:58] <MH-Discord>	 <subnekosis> do notifications update on their own or should i reload the page?
[22:45:27] <Not-2d3>	 [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp4SH
[22:45:29] <Not-2d3>	 [02miraheze/CreateWiki] 07paladox 03eaf00c3 - Add if check for $wgCreateWikiCDBRequireFiles
[22:46:46] <Not-2d3>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fp4SF
[22:46:48] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 03ad80abc - Update CW
[22:48:53] <JohnLewis>	 I mean yes... but that shouldn't be active
[22:49:01] <JohnLewis>	 why did you set the CDB config?
[22:49:24] <JohnLewis>	 at no point did I give any permission to enable it
[22:50:25] <JohnLewis>	 I checked and all the code should be if guarded. No code at all should be active
[22:51:19] <JohnLewis>	 paladox: https://github.com/miraheze/mw-config/commit/7ae4a9fd6d5b6d2cabe9f94194e50e6ed8b01ea6 revert immediately
[22:51:20] <ZppixBot>	 Title: [ Set wgCreateWikiCDBDirectory · miraheze/mw-config@7ae4a9f · GitHub ] - github.com
[22:51:29] <JohnLewis>	 otherwise wikis WILL be broken
[22:51:38] <paladox>	 JohnLewis i didn't know that
[22:51:45] <paladox>	 i was going on what the logs was saying
[22:52:18] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp49s
[22:52:20] <Not-2d3>	 [02miraheze/mw-config] 07paladox 03ee1d777 - Revert "Set wgCreateWikiCDBDirectory" This reverts commit 7ae4a9fd6d5b6d2cabe9f94194e50e6ed8b01ea6.
[22:52:25] <JohnLewis>	 at 44 you caused the outage, then reverted my changes blaming me
[22:54:00] <paladox>	 JohnLewis yes, which was a mistake by me when i realised that it started failing with scribunto errors i knew it was something else (caused by me switching that flap).
[22:54:06] <paladox>	 JohnLewis your change broke CW
[22:54:20] <paladox>	 wikis that were created in the last 24 hours were left with a broken db
[22:54:31] <paladox>	 leading to errors when users were creating accounts.
[22:54:55] <Not-2d3>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp49l
[22:54:57] <Not-2d3>	 [02miraheze/CreateWiki] 07JohnFLewis 030fd6b67 - guard stray CDB code
[22:55:17] <JohnLewis>	 my change did, but yours broke the wikis
[22:55:43] <JohnLewis>	 mine could have been found easily by looking at the errors
[22:56:00] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]  
[22:56:24] <JohnLewis>	 please deploy the above
[22:56:41] <paladox>	 ok
[22:57:37] <Not-2d3>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fp49o
[22:57:38] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 03e39d99f - Update CW
[22:58:49] <JohnLewis>	 I'm just annoyed that at this org, yet again, errors are blamed on poor code and not the poor debugging technique
[23:09:05] <Not-2d3>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fp4Ha
[23:09:06] <Not-2d3>	 [02miraheze/mediawiki] 07paladox 0324e8087 - Update CW
[23:35:30] <ZppixBot>	 Hi! Here is the list of currently open high priority tasks on Phabricator
[23:35:36] <ZppixBot>	 No updates for 3 days - https://phabricator.miraheze.org/T3812 - guiasdobrasil.com.br proxied by Cloudflare - authored by revi, assigned to None
[23:47:03] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures