[01:28:45] <Not-5be1>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fje1q
[01:28:46] <Not-5be1>	 [02miraheze/puppet] 07paladox 03e95bfde - Remove cp5 from nginx template
[01:35:26] <icinga-miraheze>	 PROBLEM - cp3 HTTPS on cp3 is CRITICAL: connect to address 128.199.139.216 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[01:35:32] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[nginx]
[01:35:50] <paladox>	 hmm
[01:36:50] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[01:37:07] <paladox>	 im checking ^^
[01:37:11] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb
[01:39:08] <paladox>	 !log reboot cp3
[01:39:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:40:50] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[01:41:11] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[01:41:32] <icinga-miraheze>	 RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 23751 bytes in 1.344 second response time
[01:41:36] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:45:50] <Not-5be1>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fje1l
[01:45:51] <Not-5be1>	 [02miraheze/puppet] 07paladox 039bccae6 - increase open files limit
[01:50:50] <JohnLewis>	 please put that in the if
[01:51:23] <Not-5be1>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fje1B
[01:51:24] <paladox>	 JohnLewis done
[01:51:24] <Not-5be1>	 [02miraheze/puppet] 07paladox 03e5aa965 - Update init.pp
[01:54:34] <paladox>	 !log set manage_etc_hosts to false in  /etc/cloud/templates/hosts.tmpl on cp3
[01:54:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[10:07:33] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs]
[10:13:32] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[10:38:44] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs]
[10:40:28] <RobertSterbal>	 Hi
[10:40:56] <RobertSterbal>	 I have a question about tiers of storage and the likelihood we will have continued storage pressure.
[10:46:53] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:10:56] <icinga-miraheze>	 PROBLEM - cp5 Disk Space on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[12:12:05] <icinga-miraheze>	 RECOVERY - cp5 Disk Space on cp5 is OK: DISK OK - free space: / 5827 MB (24% inode=94%);
[12:22:42] <RobertSterbal>	 Hi John
[12:23:02] <RobertSterbal>	 is there a link to where the communication about the storage issue is happening?
[12:23:20] <RobertSterbal>	 or is just the notice all that is being documented?
[12:36:04] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: 07F5ABF8FE84BC3736D2AAD3A2D80E04656674AE from Apt::Source puppetlabs]
[12:44:07] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:47:09] <JohnLewis>	 RobertSterbal: I have no idea, I’m not handling or dealing with it
[13:05:00] <paladox>	 RobertSterbal hi, we only have that notice. We do have this task https://phabricator.miraheze.org/T4197
[13:05:01] <ZppixBot>	 Title: [ ⚓ T4197 File Storage is running low on storage ] - phabricator.miraheze.org
[14:52:08] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 303
[14:54:03] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[20:11:57] <ZppixBot>	 Hello GiuseppeC! If you have any questions feel free to ask and someone should answer soon.
[20:14:18] <GiuseppeC>	 Hi, I have a wiki hosted here and I wish to have a  database of links with description/categories and a search field to query it and have the result. Is there any extension to do this ?
[21:20:12] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjveK
[21:20:13] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot 030f6e8a4 - BOT: Updating services config for wikis
[21:20:20] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 383
[23:07:25] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[23:15:11] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjvff
[23:15:12] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot 03066b985 - BOT: Updating services config for wikis
[23:47:05] <Not-5be1>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjvfC
[23:47:07] <Not-5be1>	 [02miraheze/mw-config] 07paladox 0348a23e7 - Switch allthetropeswiki to elasticsearch