[00:30:50] <hispano76>	 Voidwalker: I have translated the new version of the centralnotice, you can publish it please
[00:31:17] <Voidwalker>	 sure
[00:36:34] <hispano76>	 thanks Voidwalker 
[00:46:19] <Voidwalker>	 oh snap, I forgot to add flood :P
[01:06:05] <Zppix>	 Nice Voidwalker
[01:56:58] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static]
[06:15:07] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins]
[06:23:02] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:26:24] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3083 MB (12% inode=94%);
[13:16:53] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2642 MB (10% inode=94%);
[13:59:44] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvLf6
[13:59:45] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 0397d7fde - add wiki.ostans.com cert
[14:09:30] <Not-5a0a>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvLfh
[14:09:31] <Not-5a0a>	 [02miraheze/dns] 07Reception123 03359aed6 - add yukesh.com.np zone
[14:10:45] <travis-ci>	 miraheze/dns/master/359aed6 - Reception123 The build has errored. https://travis-ci.org/miraheze/dns/builds/639964804
[14:13:50] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvLJI
[14:13:51] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 0325fee78 - add yukesh.com.np cert
[14:16:26] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.ostans.com_private]
[14:16:58] <icinga-miraheze>	 PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.ostans.com_private]
[14:17:04] <Reception123>	 ack ^ 
[14:17:37] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.ostans.com_private]
[14:18:13] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.ostans.com_private]
[14:22:11] <icinga-miraheze>	 RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[14:22:33] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[14:22:33] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[14:23:34] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:25:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvLJ4
[14:25:12] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 039ebf377 - BOT: Updating services config for wikis
[14:33:16] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki /home/reception/spectrobes_pages_full.xml on mw3
[14:33:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:37:05] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=graalmilitarywiki /home/reception/graalmilitary_pages_full.xml on mw3
[14:37:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:44:15] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JvLUn
[14:44:17] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 03dacd612 - remove wiki.hyperborian.org cert no longer pointing to us
[14:44:18] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvLUc
[14:44:20] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 0333abcc5 - remove wiki.hyperborian.org cert no longer pointing to us
[14:46:43] <Reception123>	 !log manually remove custom domain for hypewiki
[14:47:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:47:09] <Reception123>	 !log PURGE BINARY LOGS before '2020-01-20 22:00:00' on db4
[14:47:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:53:25] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wiesepediawiki /home/reception/Wiesepedia-Datadump[0,A-D] on mw1
[14:53:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:02:02] <RhinosF1>	 paladox: Could not open lock file for "mwstore://local-backend/local-public/a/a6/Wikicounts-reqlive.png". Make sure your upload directory is configured correctly and your web server has permission to write to that directory. See https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:$wgUploadDirectory for more information.
[16:02:05] <ZppixBot>	 [ Manual:$wgUploadDirectory - MediaWiki ] - www.mediawiki.org
[16:02:08] <RhinosF1>	 Reception123: ^
[16:03:46] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.31, 6.01, 3.85
[16:05:14] <Reception123>	 hm
[16:06:48] <Reception123>	 images are paladox 's thing
[16:07:07] <RhinosF1>	 Reception123: paladox can't have created the folder when they setup img_auth.php
[16:13:30] <Reception123>	 paladox: hm? what's this got to do with img_auth?
[16:14:29] <JohnLewis>	 img_auth.php puts it in a non-web readable directory
[16:14:36] <JohnLewis>	 static.mh.o by definition is web readable
[16:16:30] <RhinosF1>	 JohnLewis: The wiki is set up to use img_auth.php but can't upload files currently. That sounds like the directory isn't there.
[16:16:42] <RhinosF1>	 it's stateof.miraheze.org
[16:18:42] <paladox>	 I moved his dir to private/
[16:19:22] <RhinosF1>	 paladox: but didn't create the dir
[16:19:37] <RhinosF1>	 or let mediawiki access it
[16:19:44] <Reception123>	 paladox: hi. since you're here was going to ask if we have any disk space issues with lizardfs? since I've got a 1.35 zipped file to import so it's going to be a lot
[16:20:17] <JohnLewis>	 Reception123: df -h :)
[16:21:02] <Reception123>	 JohnLewis: I know :P but it's always confusing on lizardfs since I never know if the space is total or for one of the servers that's always confusing 
[16:21:39] <paladox>	 https://github.com/miraheze/mw-config/blob/master/LocalWiki.php#L311
[16:21:40] <paladox>	 All correct to Me 
[16:21:41] <paladox>	 Reception123: no!
[16:21:42] <paladox>	 There is plenty of room
[16:21:43] <paladox>	 There is a dir
[16:21:44] <paladox>	 I’m pretty certain
[16:21:45] <ZppixBot>	 [ mw-config/LocalWiki.php at master · miraheze/mw-config · GitHub ] - github.com
[16:21:52] <paladox>	 Reception123: can you check for the dir please? :)
[16:22:05] <Reception123>	 paladox: ok, I guess while we're here can you remind me if df -h on a lizardfs server shows total space or just for that server?
[16:22:13] <Reception123>	 paladox: so it should be under /srv/mediawiki-static/private? 
[16:22:14] <RhinosF1>	 paladox: please check and make sure mediawiki can read/write to it as needed
[16:22:21] <RhinosF1>	 or Reception123 
[16:22:35] <JohnLewis>	 Reception123: df -h on any mw* servers shows the total available across the lizardfs cluster
[16:22:52] <Reception123>	 JohnLewis: ohh, is that new? I don't remember it doing that before
[16:22:57] <Reception123>	 600gb wow!
[16:23:04] <RhinosF1>	 JohnLewis: check your PMs
[16:23:15] <Reception123>	 and that's used, we've got 1.2T now that's a bit excessive
[16:23:42] <Reception123>	 drwxr-xr-x    4 www-data www-data   4096 Jan 20 21:11 stateofwiki
[16:23:56] <Reception123>	 ^ RhinosF1 there doesn't seem to be a problem with the permissions
[16:24:26] <paladox>	 No that’s not new
[16:24:26] <paladox>	 That’s what mount does :)
[16:24:43] <Reception123>	 paladox: why do we have so much free space though?
[16:24:57] <Reception123>	 I doubt we'll need 1.2TB worth of space for another year
[16:25:01] <paladox>	 Because it’s 2tb
[16:42:29] <paladox>	 RhinosF1 fixed
[16:42:43] <Reception123>	 paladox: what was the problem?
[16:43:00] <paladox>	 mount disconnected on mw2, so had to umount *
[16:43:43] <paladox>	 i did * just because i couldn't be botherd to type mediawiki-static :P
[16:44:13] <RhinosF1>	 Reception123: can you upload them two files I PM'd?
[16:44:23] <Reception123>	 ok
[16:49:46] <Reception123>	 RhinosF1: done
[16:51:40] <Operator873>	 Reception123 1 Yb would be excessive
[16:51:42] <Operator873>	 :-)
[16:52:40] <Reception123>	 Operator873: lol 
[16:54:06] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:53:44] <RhinosF1>	 Operator873: is Operator873 on MH you?
[17:56:55] <Operator873>	 indeed it is
[17:57:06] <RhinosF1>	 Operator873: good
[19:08:05] <ZppixBot>	 Hello musicalfriend[m]! If you have any questions, feel free to ask and someone should answer soon.
[19:08:05] <musicalfriend[m]>	 sorry friends i was wrong today is trump impeachement holiday so today i must sing to you! Zppix Reception123 RhinosF1 Voidwalker
[19:08:10] <musicalfriend[m]>	 With so many light years to go and things to be found
[19:08:15] <musicalfriend[m]>	 It's the final countdown
[19:08:15] <musicalfriend[m]>	 The final countdown
[19:08:22] <musicalfriend[m]>	 Ohh ho ohh
[19:08:22] <musicalfriend[m]>	 The final countdown, oh ho
[19:08:27] <musicalfriend[m]>	 It's the final countdown
[19:08:27] <musicalfriend[m]>	 We're leaving together
[19:08:27] <musicalfriend[m]>	 The final countdown
[19:15:18] <Zppix>	 grumble: niko any updates on getting matrix to do something about our friend?
[19:15:58] <grumble>	 we're in touch with them 
[19:16:27] <Zppix>	 I know that but are they able to do anything? grumble
[19:16:36] <Reception123>	 like maybe block the IP?
[19:17:21] <Zppix>	 I mean if they need our cooperation they can contact us we’re more then willing to provide whatever details we can grumble
[19:18:20] <grumble>	 nod, i'll forward that to them verbatim if you don't mind :)
[19:18:36] <Zppix>	 Go ahead
[19:18:59] <Zppix>	 Our staff email is on site if they want to use that as well
[19:19:16] <grumble>	 can you just gimme a link i can copy?
[19:20:04] <Zppix>	 grumble: our email is tech[at]miraheze.org or if they want someone directly to contact my email is zppix[at]miraheze.org
[19:31:53] * hispano76 greetings
[19:48:47] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.46, 0.53, 1.60
[19:54:36] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 16.70, 7.97, 4.07
[20:09:30] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.96, 0.68, 1.74
[20:12:06] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.57, 0.72, 1.46
[20:50:47] <Operator873>	 Zppix you have a minute?
[20:51:27] <Operator873>	 PM when you can
[20:52:01] <RhinosF1>	 Operator873: I think Zppix went for the day, anything we can help with?
[21:09:35] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.65, 2.45, 2.17
[21:12:35] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.51, 1.46, 1.84
[21:15:30] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.23, 0.65, 1.36
[21:29:36] <paladox>	 !log restart nutcracker on mw1
[21:29:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:39:23] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.89, 2.41, 2.27
[21:44:32] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.80, 1.31, 1.76
[21:46:55] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.70, 1.12, 1.58
[22:11:59] <Zppix>	 RhinosF1: handled the pm with operator
[23:06:12] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:23:47] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures