[00:04:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.85, 20.83, 14.32
[00:06:29] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.99, 24.14, 16.40
[00:08:29] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.99, 19.54, 15.68
[00:14:52] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 15.52, 21.26, 17.94
[00:16:53] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 9.24, 17.10, 16.83
[05:20:20] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 pushed 033 commits to 03tou-pp [+0/-0/±3] 13https://git.io/JJkC9
[05:20:21] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 03bbd9762 - Merge branch 'master' into tou-pp
[05:20:23] <Not-6258>	 [02bots-web] 07RhinosF1 synchronize pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[06:20:28] <Not-6258>	 [02mw-config] 07Reception123 closed pull request 03#3132: Misc. config for dcwiki T5838 - 13https://git.io/JJfJG
[06:20:30] <Not-6258>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkl0
[06:20:31] <Not-6258>	 [02miraheze/mw-config] 07Amanda-Catherine 031cbf639 - Misc. config for dcwiki T5838 (#3132) * Misc. config for dcwiki T5838 * Update LocalSettings.php * remove extra space * Update LocalSettings.php
[06:44:18] <icinga-miraheze>	 PROBLEM - mw6 Disk Space on mw6 is WARNING: DISK WARNING - free space: / 2004 MB (10% inode=68%);
[07:10:33] <Reception123>	 !log /usr/bin/nice -19 /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/generateMirahezeSitemap.php
[07:10:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:21:01] <icinga-miraheze>	 PROBLEM - mw7 Disk Space on mw7 is WARNING: DISK WARNING - free space: / 2007 MB (10% inode=68%);
[07:21:14] <icinga-miraheze>	 PROBLEM - mw6 Disk Space on mw6 is CRITICAL: DISK CRITICAL - free space: / 1129 MB (5% inode=68%);
[07:45:45] <icinga-miraheze>	 PROBLEM - mw4 Disk Space on mw4 is WARNING: DISK WARNING - free space: / 2027 MB (10% inode=68%);
[07:47:42] <icinga-miraheze>	 PROBLEM - mw5 Disk Space on mw5 is WARNING: DISK WARNING - free space: / 2020 MB (10% inode=68%);
[08:58:27] <icinga-miraheze>	 PROBLEM - mw7 Disk Space on mw7 is CRITICAL: DISK CRITICAL - free space: / 1136 MB (5% inode=68%);
[08:58:42] <RhinosF1>	 Reception123: ^
[08:59:12] <Reception123>	 RhinosF1: I wonder what's causing that 
[09:00:28] <RhinosF1>	 Reception123: look?
[09:01:34] <RhinosF1>	 https://unix.stackexchange.com/questions/125429/tracking-down-where-disk-space-has-gone-on-linux
[09:01:35] <MirahezeBot>	 [ command line - Tracking down where disk space has gone on Linux? - Unix & Linux Stack Exchange ] - unix.stackexchange.com
[09:12:44] <RhinosF1>	 Reception123: ?
[09:13:01] <Reception123>	 RhinosF1: didn't find anything yet
[09:13:15] <RhinosF1>	 ok
[09:13:24] <Reception123>	 RhinosF1: if you have access maybe you can help out?
[09:13:34] <Reception123>	 it's mostly a guessing game trying out all the dirs with du 
[09:13:47] <RhinosF1>	 Ok
[09:19:27] <RhinosF1>	 Reception123: no access
[09:19:51] <Reception123>	 RhinosF1: still haven't found what would be taking so much space, just normal sizes 
[09:21:15] <RhinosF1>	 Reception123: strange
[09:22:21] <Reception123>	 RhinosF1: I guess we could delete /tmp? (3.6 GB)
[09:22:55] <RhinosF1>	 Reception123: you can try, maybe zip logs up as well
[09:23:09] <Reception123>	 well /tmp should at least solve the problem for now 
[09:23:30] * RhinosF1 is going in 7 mins - cc paladox
[09:23:37] <RhinosF1>	 Reception123: ok
[09:23:48] <icinga-miraheze>	 RECOVERY - mw6 Disk Space on mw6 is OK: DISK OK - free space: / 4627 MB (24% inode=68%);
[09:24:06] <icinga-miraheze>	 RECOVERY - mw5 Disk Space on mw5 is OK: DISK OK - free space: / 4624 MB (24% inode=68%);
[09:24:13] <Reception123>	 !log cleared /tmp on mw[47]
[09:24:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:24:39] <RhinosF1>	 Why did mw[56] go ok then?
[09:24:48] <icinga-miraheze>	 RECOVERY - mw7 Disk Space on mw7 is OK: DISK OK - free space: / 4699 MB (24% inode=68%);
[09:24:49] <Reception123>	 RhinosF1: because I cleared /tmp...
[09:25:12] <RhinosF1>	 Reception123: you meant [4-7] then not [47]
[09:25:20] <Reception123>	 oh yes
[09:25:26] <Reception123>	 my bad
[09:25:27] <RhinosF1>	 Reception123: backends are dropping off
[09:25:40] <icinga-miraheze>	 RECOVERY - mw4 Disk Space on mw4 is OK: DISK OK - free space: / 4314 MB (22% inode=68%);
[09:25:41] <icinga-miraheze>	 PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 3 backends are down. mw5 mw6 mw7
[09:25:43] <RhinosF1>	 !log <correction> [4-7] not [47]
[09:25:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:26:03] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 3 backends are down. mw5 mw6 mw7
[09:26:13] <Reception123>	 RhinosF1: that makes no sense
[09:26:20] <RhinosF1>	 Reception123: only mw4 is now alive
[09:26:51] <RhinosF1>	 Reception123: what the hell is in /tmp
[09:27:04] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw5 mw6 mw7
[09:27:08] <Reception123>	 RhinosF1: I don't know but I used to delete /tmp countless times a while back when it didn't delete itself
[09:27:18] <icinga-miraheze>	 PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 3 backends are down. mw5 mw6 mw7
[09:27:24] <Reception123>	 normally it deletes itself 
[09:27:44] <RhinosF1>	 Reception123: try forcing a puppet run and see if it rebuilds whatever has gone crappy???
[09:27:49] * RhinosF1 needs to go
[09:27:54] <Reception123>	 https://www.irccloud.com/pastebin/gswvMs9c/
[09:27:54] <MirahezeBot>	 [ Snippet | IRCCloud ] - www.irccloud.com
[09:28:36] <RhinosF1>	 Reception123: f f f
[09:28:52] <Reception123>	 think I fixed it
[09:28:59] <Reception123>	 !log recreate /tmp on mw[567]
[09:29:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:29:19] <RhinosF1>	 Reception123: force puppet again - that should work
[09:32:10] <icinga-miraheze>	 PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.31, 6.48, 3.57
[09:34:11] <icinga-miraheze>	 RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.03, 6.09, 3.82
[09:39:20] <icinga-miraheze>	 RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy
[09:39:31] <Reception123>	 !log sudo chmod 777 /tmp on mw[57]
[09:39:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:39:39] <icinga-miraheze>	 RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy
[09:40:22] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy
[09:41:16] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy
[09:55:12] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJk0n
[09:55:14] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot 03bfcde03 - BOT: Updating services config for wikis
[10:08:00] <RhinosF1>	 Reception123: back we look alive
[10:08:08] <Reception123>	 RhinosF1: yeah I figured it out
[10:08:16] <Reception123>	 RhinosF1: had to recreate /tmp and chmod 777
[10:08:20] <RhinosF1>	 Ok
[11:11:39] <MH-Discord>	 <Zppix> Was i pinged?
[11:12:23] <Reception123>	 @Zppix don't think so
[11:12:33] <Reception123>	 maybe from -feed when I supported your GS request?
[11:12:54] <MH-Discord>	 <Zppix> It was from here according to discord
[11:13:04] <MH-Discord>	 <Zppix> The relay now auto pings if username match
[11:13:32] <Reception123>	 ah you're right it was RhinosF1 
[11:13:35] <Reception123>	 RhinosF1> Zppix: I can't get the css to fix the footer to the end of the page and leave whitespace if it's too small to take up a whole screen
[11:15:01] <MH-Discord>	 <Zppix> Yeah idk
[11:15:09] <MH-Discord>	 <Zppix> Css isnt my expertise
[11:21:55] <RhinosF1>	 I think I found something to try
[13:04:49] <MH-Discord>	 <Avengium (Ángel)> hi
[13:40:11] <Zppix>	 hi
[13:41:44] <Zppix>	 paladox:  we are intermitten 503s
[13:42:48] <paladox>	 !log depool mw 4,5,6,7
[13:42:52] <paladox>	 !log and repool
[13:42:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:42:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:43:25] <Zppix>	 paladox:  any idea why its taking so long for me to login
[13:45:03] <Not-6258>	 [02miraheze/ssl] 07Pix1234 created branch 03T5868 13https://git.io/JJkaB
[13:45:04] <Not-6258>	 [02ssl] 07Pix1234 created branch 03T5868 - 13https://git.io/vxP9L
[13:46:42] <paladox>	 based on grafana we haven't gone down, people may get errors if they upload and it goes past the timeout
[13:46:51] <paladox>	 (not even itermittent 503s)
[13:50:00] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:51:03] <Not-6258>	 [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03T5868 [+1/-0/±0] 13https://git.io/JJkai
[13:51:04] <Not-6258>	 [02miraheze/ssl] 07Pix1234 036b68478 - Create minetestserver-cc.ddns.net.crt
[13:52:02] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Sun 16 Aug 2020 23:10:01 GMT +0000.
[13:52:30] <Not-6258>	 [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03T5868 [+0/-0/±1] 13https://git.io/JJkab
[13:52:32] <Not-6258>	 [02miraheze/ssl] 07Pix1234 03af40ddd - + minetestserver-cc.ddns.net
[13:52:46] <Not-6258>	 [02ssl] 07Pix1234 opened pull request 03#332: T5868 - 13https://git.io/JJkaN
[13:53:10] <Not-6258>	 [02ssl] 07Pix1234 closed pull request 03#332: T5868 - 13https://git.io/JJkaN
[13:53:11] <Not-6258>	 [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJkaA
[13:53:13] <Not-6258>	 [02miraheze/ssl] 07Pix1234 03fea3393 - T5868 +minetestccserverwiki ssl (#332) * Create minetestserver-cc.ddns.net.crt * + minetestserver-cc.ddns.net
[13:53:14] <Not-6258>	 [02ssl] 07Pix1234 deleted branch 03T5868 - 13https://git.io/vxP9L
[13:53:16] <Not-6258>	 [02miraheze/ssl] 07Pix1234 deleted branch 03T5868
[13:53:49] <Reception123>	 Zppix: thanks for helping out with certs :)
[13:54:08] <Zppix>	 Reception123:  i may have to leave for work before puppet runs so if i dont set the url in managewiki can you
[13:54:17] <Zppix>	 (i have to leave in 5 mins
[13:54:57] <Reception123>	 Zppix: sure 
[13:55:02] <Zppix>	 ok ty
[13:55:10] <Zppix>	 (i didnt realize what time it was when i did it)
[15:39:40] <paladox>	 !log deleted redis keys for nynthidbwiki & cleared objectcache table
[15:39:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:17:22] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.11, 19.42, 14.13
[16:18:33] <paladox>	 !log echo "" > php-error.log on mw 4,5,6,7 (file size is 3gb plus)
[16:18:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:19:25] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 10.05, 15.99, 13.56
[17:28:45] <Not-6258>	 [02bots-web] 07RhinosF1 assigned pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[17:28:58] <Not-6258>	 [02bots-web] 07RhinosF1 labeled pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[17:40:21] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJk1e
[17:40:23] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot 03274a29e - BOT: Updating services config for wikis
[18:42:19] <MH-Discord>	 <Avengium (Ángel)> Void, how is going the request for adoption?
[18:57:17] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 4.08, 2.44, 1.40
[19:03:18] <Voidwalker>	 Ah, thanks for the reminder, I'll go through those soon
[19:05:17] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.08, 1.67, 1.47
[19:43:27] <MH-Discord>	 <Avengium (Ángel)> good
[19:43:59] <Voidwalker>	 Should be just about all processed
[19:44:48] <MH-Discord>	 <Avengium (Ángel)> i have a doubt about Disqus. i created an account to submit my wiki, but is disqus a paid service? https://https-tuscriaturas-miraheze-org.disqus.com/admin/install/subscription/
[19:44:49] <MirahezeBot>	 [ - Disqus ] - https-tuscriaturas-miraheze-org.disqus.com
[19:47:53] <MH-Discord>	 <Avengium (Ángel)> if i have to pay for disqus, then i can't use it.
[21:03:05] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.54, 17.37, 13.71
[21:05:06] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.83, 15.95, 13.59
[21:25:05] <RhinosF1>	 @Avengium: No, we have enough to add it.
[21:32:46] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 pushed 035 commits to 03tou-pp [+0/-0/±7] 13https://git.io/JJk7q
[21:32:47] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 033e521ce - fix
[21:32:49] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 031a14e60 - Merge branch 'tou-pp' of https://github.com/miraheze/bots-web into tou-pp merge
[21:32:50] <MirahezeBot>	 [ GitHub - miraheze/bots-web: Source code for the static bot documentation ] - github.com
[21:32:50] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 032d72a44 - Merge branch 'tou-pp' of https://github.com/miraheze/bots-web into tou-pp
[21:32:51] <MirahezeBot>	 [ GitHub - miraheze/bots-web: Source code for the static bot documentation ] - github.com
[21:32:52] <Not-6258>	 [02miraheze/bots-web] ... and 2 more commits.
[21:32:53] <Not-6258>	 [02bots-web] 07RhinosF1 synchronize pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[21:33:29] <RhinosF1>	 Zppix: there's that ^ but the tou/pp bar moves as you scroll
[21:38:12] <revi>	 BTW if someone is interested https://en.citizendium.org/wiki?title=Welcome_to_Citizendium&diff=100864611&oldid=100857010
[21:38:16] <MirahezeBot>	 [ Difference between revisions of "Welcome to Citizendium" - encyclopedia article - Citizendium ] - en.citizendium.org
[21:45:06] <RhinosF1>	 revi: ty
[21:49:16] <RhinosF1>	 Zppix: I've done it!
[21:50:04] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 pushed 031 commit to 03tou-pp [+0/-0/±1] 13https://git.io/JJk76
[21:50:06] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 03c710f41 - THIS IS IT
[21:50:07] <Not-6258>	 [02bots-web] 07RhinosF1 synchronize pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[21:50:46] * RhinosF1 waits for travis
[21:52:01] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 pushed 031 commit to 03tou-pp [+0/-0/±1] 13https://git.io/JJk7M
[21:52:02] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 03654082d - travis
[21:52:04] <Not-6258>	 [02bots-web] 07RhinosF1 synchronize pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[21:54:12] <Not-6258>	 [02bots-web] 07RhinosF1 closed pull request 03#10: Add ToU/PP - 13https://git.io/JJkIo
[21:54:14] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 pushed 031 commit to 03master [+2/-0/±5] 13https://git.io/JJk7S
[21:54:15] <Not-6258>	 [02miraheze/bots-web] 07RhinosF1 035d9b6d3 - Add ToU/PP (#10) Co-authored-by: MacFan4000 <paulfkeffer@gmail.com>
[22:30:20] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJk5N
[22:30:21] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot 032be6824 - BOT: Updating services config for wikis
[22:30:53] <RhinosF1>	 relay was me
[23:15:13] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJkF0
[23:15:14] <Not-6258>	 [02miraheze/services] 07MirahezeSSLBot 033fdadab - BOT: Updating services config for wikis
[23:29:52] <SPF|Cloud>	 !log install sysstat on db9
[23:29:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log