[00:02:38] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvTnn
[00:02:39] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03a593421 - matomo: Update to 3.13.1
[00:03:33] <Examknow>	   Voidwalker: There is something cool with my bot that you should see
[00:03:57] <Examknow>	 it would be pretty handy for miraheze
[00:12:12] <paladox>	 !log apt-get upgrade - misc2
[00:12:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:26:15] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis]
[00:32:39] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[01:43:10] <paladox>	 !log too long, going to manually update
[01:43:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:48:35] <paladox>	 !log restoring mw_permissions from a newish backup (after running "delete from mw_permissions where perm_group = "\*";" which seemed to updated 4,000 results)
[01:48:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:04:58] <ZppixBot>	 Hello Wiki-1776! If you have any questions, feel free to ask and someone should answer soon.
[02:11:46] <paladox>	 !log rm -rf /srv/mediawiki/w/cache/manage*/** on mw* and lizardfs6
[02:11:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[06:27:46] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3105 MB (12% inode=94%);
[07:22:04] <ZppixBot>	 Hello MirahezeLogbot! If you have any questions, feel free to ask and someone should answer soon.
[15:35:16] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%);
[16:03:55] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:07:35] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:07:54] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:14:06] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 5.417 second response time
[16:14:33] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:16:07] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[16:20:13] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[16:24:10] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[16:57:36] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvT1C
[16:57:37] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03dc57e2d - Fix wgEnableScaryTranscluding default value Bug: T5110
[18:10:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvTDw
[18:10:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 0316b0374 - BOT: Updating services config for wikis
[18:18:48] <RhinosF1>	 .uptime
[18:18:48] <ZppixBot>	 I've been sitting here for 3 days, 3:51:35 and I keep going!
[18:42:33] * k6ka unplugs ZppixBot
[18:43:13] * RhinosF1 doesn’t need k6ka to break anything. I’m deploying today.
[18:43:58] <k6ka>	 Good
[18:44:21] <k6ka>	 You're going to rm -rf * everything on the servers?
[18:44:34] <k6ka>	 That would resolve all of our storage space problems
[18:44:45] <RhinosF1>	 k6ka: just mess with welcome database for ZppixBot
[18:44:54] <RhinosF1>	 I cant break MH anymore
[18:45:11] <k6ka>	 Oh, so you're going to make ZppixBot annoy me every day again
[18:45:18] <k6ka>	 oh well, could've been worse
[18:45:31] <k6ka>	 rm -rf the database daily
[18:45:54] <RhinosF1>	 k6ka: I plan to make it able to detect users better
[18:46:40] <paladox>	 k6ka: storage? :P
[18:46:41] <paladox>	 We have 2tb for file storage
[18:46:42] <paladox>	 Just the dbs are running out of space
[18:46:43] <paladox>	 Well more like 1.3tb left? Or 1.5
[18:47:25] <k6ka>	 We can always break into Wikia's datacenters and steal one of their machines
[18:47:54] <k6ka>	 What are they planning to use it for, furries? :)
[18:48:04] <k6ka>	 Definitely nothing useful or beneficial
[18:48:49] <RhinosF1>	 k6ka: no we borrow on an indefinite deal
[18:48:53] <paladox>	 Lol
[18:48:54] <paladox>	 Hahaha
[18:48:55] <RhinosF1>	 But dont say
[18:51:10] <JohnLewis>	 one machine would hardly solve problems :P
[19:10:49] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thefinalrumblewiki /home/reception/thefinalrumble_pages_full.xml
[19:10:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:16:53] <ZppixBot>	 Hello Hispano! If you have any questions, feel free to ask and someone should answer soon.
[19:19:28] <ZppixBot>	 Hello Hispano! If you have any questions, feel free to ask and someone should answer soon.
[19:35:34] <paladox>	 JohnLewis lol
[20:02:07] <ZppixBot>	 Hello sario528! If you have any questions, feel free to ask and someone should answer soon.
[21:26:19] <RhinosF1>	 JohnLewis, Voidwalker: can you delete those wikis 8ight requested on SN?