[00:02:42] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3697 MB (19% inode=93%);
[00:10:02] <icinga-miraheze>	 PROBLEM - wiki.hanaoto.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:12:07] <icinga-miraheze>	 RECOVERY - wiki.hanaoto.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hanaoto.me' will expire on Mon 06 Apr 2020 09:13:25 AM GMT +0000.
[00:16:26] <icinga-miraheze>	 PROBLEM - wiki.hanaoto.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:45:40] <paladox>	 !log drop large wiki drops on db4/db5
[00:45:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:49:42] <icinga-miraheze>	 RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 103370 MB (28% inode=95%);
[01:08:04] <icinga-miraheze>	 RECOVERY - wiki.hanaoto.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hanaoto.me' will expire on Mon 06 Apr 2020 09:13:25 AM GMT +0000.
[01:34:26] <paladox>	 !log MariaDB [simswiki]> truncate objectcache; - db6
[01:34:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[02:58:30] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:58:45] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:59:08] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 3 backends are down. mw5 mw6 mw7
[02:59:15] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw6 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:59:15] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw5 mw6 mw7
[02:59:42] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw6 mw7
[02:59:57] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw7 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[03:00:35] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[03:00:52] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[03:01:06] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.094 second response time
[03:01:21] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw6 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 3.607 second response time
[03:02:44] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.761 second response time
[03:02:53] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.309 second response time
[03:03:05] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.761 second response time
[03:03:16] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy
[03:03:18] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy
[03:03:44] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 10 backends are healthy
[03:04:16] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw7 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.079 second response time
[06:27:25] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3269 MB (13% inode=94%);
[06:35:21] <Reception123>	 !log deleted and dropped streamelementswiki (T5307)
[06:35:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[06:44:24] <Not-a789>	 [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvwFF
[06:44:24] <ZppixBot>	 Hello Not-a789! If you have any questions, feel free to ask and someone should answer soon.
[06:44:26] <Not-a789>	 [02miraheze/MirahezeMagic] 07Reception123 03ef50e93 - remove formatting as form doesn't accept html or wikitext
[07:27:40] <Not-a789>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvwAG
[07:27:41] <Not-a789>	 [02miraheze/dns] 07Reception123 035f29528 - add rivworld.wiki zone
[07:29:02] <RhinosF1>	 morning Reception123 
[07:29:13] <Reception123>	 hi RhinosF1 
[12:12:43] <icinga-miraheze>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:14:38] <icinga-miraheze>	 RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19671 bytes in 0.852 second response time
[13:37:06] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrGy
[13:37:08] <Not-a789>	 [02miraheze/mw-config] 07paladox 035148013 - Fix default value for wmgWebChatClient
[13:37:09] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[13:37:11] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2932: Fix default value for wmgWebChatClient - 13https://git.io/JvrGS
[13:38:37] <travis-ci>	 miraheze/mw-config/paladox-patch-1/5148013 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/659531057
[13:39:39] <Not-a789>	 [02mw-config] 07paladox synchronize pull request 03#2932: Fix default value for wmgWebChatClient - 13https://git.io/JvrGS
[13:39:40] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrGx
[13:39:42] <Not-a789>	 [02miraheze/mw-config] 07paladox 03dd66e94 - Update LocalSettings.php
[13:40:11] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2932: Fix default value for wmgWebChatClient - 13https://git.io/JvrGS
[13:40:12] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrZU
[13:40:14] <Not-a789>	 [02miraheze/mw-config] 07paladox 03ba8b156 - Fix default value for wmgWebChatClient (#2932) * Fix default value for wmgWebChatClient * Update LocalSettings.php
[13:40:15] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[13:40:17] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[13:41:13] <travis-ci>	 miraheze/mw-config/paladox-patch-1/dd66e94 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/659531496
[13:45:19] <Not-a789>	 [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk
[13:45:20] <Not-a789>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrZZ
[13:45:22] <Not-a789>	 [02miraheze/DataDump] 07paladox 03b89417c - Update extension.json
[13:53:48] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrZH
[13:53:49] <Not-a789>	 [02miraheze/dns] 07paladox 03e7eed1d - Make altversewiki use cp7
[14:06:31] <Not-a789>	 [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk
[14:06:32] <Not-a789>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrnD
[14:06:34] <Not-a789>	 [02miraheze/DataDump] 07paladox 03d3fc82e - Update SpecialDataDump.php
[14:07:25] <RhinosF1>	 Morning paladox
[14:07:49] <paladox>	 hi
[14:08:15] <travis-ci>	 miraheze/DataDump/paladox-patch-1/d3fc82e - paladox The build was broken. https://travis-ci.com/miraheze/DataDump/builds/152267564
[14:08:23] <RhinosF1>	 Well done last night, I’m really happy with how well that went
[14:09:52] <RhinosF1>	 I’m actually quite proud of SRE
[14:10:08] <RhinosF1>	 Which is unusal for me :)
[14:10:42] <Not-a789>	 [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk
[14:10:44] <Not-a789>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvrn9
[14:10:45] <Not-a789>	 [02miraheze/DataDump] 07paladox 03bb36b3b - Update SpecialDataDump.php
[14:10:58] <Not-a789>	 [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk
[14:10:59] <Not-a789>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrnQ
[14:11:01] <Not-a789>	 [02miraheze/DataDump] 07paladox 03536a53f - Update extension.json
[14:13:55] <travis-ci>	 miraheze/DataDump/paladox-patch-1/536a53f - paladox The build was fixed. https://travis-ci.com/miraheze/DataDump/builds/152267683
[14:19:53] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures
[14:28:47] <Not-a789>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvrch
[14:28:49] <Not-a789>	 [02miraheze/DataDump] 07paladox 03ef0be2a - Update extension.json
[14:28:50] <Not-a789>	 [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk
[14:30:15] <Not-a789>	 [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrCc
[14:30:17] <Not-a789>	 [02miraheze/DataDump] 07paladox 0309757c1 - Update DataDump.php
[14:30:18] <Not-a789>	 [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk
[15:18:10] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrlG
[15:18:12] <Not-a789>	 [02miraheze/dns] 07paladox 0347eeebb - Make unmusicfest use cp7
[15:20:09] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvrl8
[15:20:11] <Not-a789>	 [02miraheze/mw-config] 07paladox 03e4aa34e - Switch unmusicfestwiki to db6
[15:20:12] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[15:20:14] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2933: Switch unmusicfestwiki to db6 - 13https://git.io/Jvrl4
[15:21:16] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrlB
[15:21:17] <Not-a789>	 [02miraheze/dns] 07paladox 03973dd3d - Revert "Make unmusicfest use cp7" This reverts commit 47eeebb3219e85f9e0b72e7a0c49467db809ac97.
[15:21:19] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2933: Switch unmusicfestwiki to db6 - 13https://git.io/Jvrl4
[15:21:21] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[15:21:22] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[15:21:46] <travis-ci>	 miraheze/mw-config/paladox-patch-1/e4aa34e - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/659553866
[15:22:44] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvrlz
[15:22:46] <Not-a789>	 [02miraheze/dns] 07paladox 03607380a - Make openhatch use cp7
[15:23:03] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvrlg
[15:23:04] <Not-a789>	 [02miraheze/dns] 07paladox 034e88911 - Update miraheze.org
[15:23:51] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvrla
[15:23:52] <Not-a789>	 [02miraheze/mw-config] 07paladox 03e9c39df - Switch openhatchwiki to db6
[15:23:54] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[15:23:55] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2934: Switch openhatchwiki to db6 - 13https://git.io/Jvrlw
[15:25:49] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2934: Switch openhatchwiki to db6 - 13https://git.io/Jvrlw
[15:25:50] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvrli
[15:25:52] <Not-a789>	 [02miraheze/mw-config] 07paladox 03ce1bf76 - Switch openhatchwiki to db6 (#2934)
[15:25:53] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[15:25:55] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[15:31:54] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrlQ
[15:31:55] <Not-a789>	 [02miraheze/puppet] 07paladox 03e0cec19 - Update default-grants.sql.erb
[15:36:38] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvrlp
[15:36:39] <Not-a789>	 [02miraheze/puppet] 07paladox 03c17c681 - cp7: Check allthetropeswiki instead of loginwiki temporarily The new infrastructure seems better and latency decreases.
[15:36:41] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[15:36:42] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1279: cp7: Check allthetropeswiki instead of loginwiki temporarily - 13https://git.io/Jvrlh
[15:37:14] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1279: cp7: Check allthetropeswiki instead of loginwiki temporarily - 13https://git.io/Jvrlh
[15:37:16] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvr8e
[15:37:17] <Not-a789>	 [02miraheze/puppet] 07paladox 03f70229a - cp7: Check allthetropeswiki instead of loginwiki temporarily (#1279) The new infrastructure seems better and latency decreases.
[15:37:19] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[15:37:20] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[15:38:36] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvr8U
[15:38:38] <Not-a789>	 [02miraheze/puppet] 07paladox 03ad47c35 - Update default.vcl
[16:44:42] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrBe
[16:44:44] <Not-a789>	 [02miraheze/puppet] 07paladox 0378cd98e - bacula: Remove db5
[16:44:45] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[16:44:47] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1280: bacula: Remove db5 - 13https://git.io/JvrBv
[16:45:33] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrBf
[16:45:34] <Not-a789>	 [02miraheze/puppet] 07paladox 0309f22a5 - Update director.pp
[16:45:36] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1280: bacula: Remove db5 - 13https://git.io/JvrBv
[16:45:47] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrBJ
[16:45:48] <Not-a789>	 [02miraheze/puppet] 07paladox 03f54ae8c - Update nrpe.cfg.erb
[16:45:50] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1280: bacula: Remove db5 - 13https://git.io/JvrBv
[16:47:13] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvrBI
[16:47:15] <Not-a789>	 [02miraheze/mw-config] 07paladox 0365998dc - database: Remove db5 db5 is no longer use, remove it.
[16:47:16] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[16:47:18] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2935: database: Remove db5 - 13https://git.io/JvrBL
[16:47:26] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[16:47:27] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[17:03:38] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:10:45] <paladox>	 !log apt-get upgrade db5
[17:10:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:12:22] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:17:00] <paladox>	 !log apt-get -t stretch-backports install linux-image-amd64 - db5
[17:17:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:18:53] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2935: database: Remove db5 - 13https://git.io/JvrBL
[17:18:55] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrRw
[17:18:56] <Not-a789>	 [02miraheze/mw-config] 07paladox 032c48c2e - database: Remove db5 (#2935) db5 is no longer use, remove it.
[17:18:58] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3
[17:18:59] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1
[17:20:40] <paladox>	 !log set global table_definition_cache = 10000; - db4
[17:20:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:20:59] <paladox>	 !log MariaDB [(none)]> set global table_open_cache = 10000; - db4
[17:21:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:32:07] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%);
[17:39:43] <paladox>	 !log stopped gdnsd + purged it from misc1
[17:39:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:41:51] <paladox>	 !log removed prometheus_gdnsd_stats cron from misc1
[17:41:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:34:02] <paladox>	 !log root@cloud1:/var/lib/vz/images/118# qemu-img convert vm-118-disk-0.qcow2 vm-118-disk-0.row
[18:34:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:37:01] <paladox>	 !log root@cloud1:/var/lib/vz/images/118# mv vm-118-disk-0.row vm-118-disk-0.raw
[18:37:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:38:31] <paladox>	 !log upgrade puppet-agent - ns2
[18:38:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:21:12] <paladox>	 !log root@cloud1:/var/lib/vz/images/118# rm vm-118-disk-0.qcow2
[19:21:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:26:28] <paladox>	 !log upgrade puppet-agent on cloud[12]
[19:26:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:46:02] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:47:59] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.313 second response time
[21:24:24] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2056 MB (10% inode=93%);
[21:26:20] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw5 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:28:21] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw5 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.311 second response time
[23:23:04] <paladox>	 !log reinstalling ns1 as buster
[23:23:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:27:13] <icinga-miraheze>	 PROBLEM - ns1 Disk Space on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused
[23:27:21] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused
[23:28:00] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused
[23:28:34] <icinga-miraheze>	 PROBLEM - ns1 Current Load on ns1 is CRITICAL: connect to address 192.184.82.120 port 5666: Connection refusedconnect to host 192.184.82.120 port 5666: Connection refused
[23:29:31] <icinga-miraheze>	 PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:31:00] <SPF|Cloud>	 paladox: has ns1 been depooled?
[23:31:14] <paladox>	 SPF|Cloud depooled?
[23:31:24] <SPF|Cloud>	 Yes? Vanity records wise
[23:31:39] <SPF|Cloud>	 Did you consult JohnLewis?
[23:32:04] <paladox>	 Yup
[23:32:13] <SPF|Cloud>	 Alright 
[23:33:00] <SPF|Cloud>	 Just like databases, DNS can be stupid at times.. so touching it cannot be done lightly
[23:33:24] <paladox>	 ok, i believe the last time we just went and reinstalled it too.
[23:33:33] <paladox>	 (when we did jessie -> stretch)
[23:34:03] <icinga-miraheze>	 RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.42, 0.45, 0.35
[23:35:15] <mirahezebots_>	 PROBLEM - cp7 Stunnel Http for lizardfs6 on cp7 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8083: HTTP/1.1 200 OK
[23:35:25] <mirahezebots_>	 PROBLEM - cp7 Stunnel Http for test2 on cp7 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.001 second response time
[23:35:27] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output
[23:35:34] <icinga-miraheze>	 RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 11419 MB (89% inode=92%);
[23:35:36] <mirahezebots_>	 PROBLEM - cp6 Stunnel Http for lizardfs6 on cp6 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8083: HTTP/1.1 200 OK
[23:37:22] <mirahezebots_>	 PROBLEM - cp6 Stunnel Http for test2 on cp6 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.001 second response time
[23:37:32] <icinga-miraheze>	 RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.109 seconds response time. miraheze.org returns 51.161.32.127
[23:37:35] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:37:43] <mirahezebots_>	 RECOVERY - db6 MySQL on db6 is OK: Uptime: 185215 Threads: 30 Questions: 97618236 Slow queries: 4426109 Opens: 60196 Flush tables: 1 Open tables: 6412 Queries per second avg: 527.053
[23:39:32] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvrrf
[23:39:33] <Not-a789>	 [02miraheze/puppet] 07paladox 03f4a90cb - Update ns1.yaml
[23:39:55] <mirahezebots_>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:42:20] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:42:55] <mirahezebots_>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures