[00:01:41] <icinga-miraheze>	 PROBLEM - test1 php7.2-fpm on test1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2'
[00:05:41] <icinga-miraheze>	 RECOVERY - test1 php7.2-fpm on test1 is OK: PROCS OK: 7 processes with command name 'php-fpm7.2'
[01:39:35] <paladox>	 !log rebooted misc2 as redis is unresponsive
[01:40:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[06:34:02] <Not-8d6a>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJKa
[06:34:03] <Not-8d6a>	 [02miraheze/dns] 07Reception123 03de45dad - add "ndg" subdomain
[06:45:37] <Not-8d6a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fhJKP
[06:45:39] <Not-8d6a>	 [02miraheze/ssl] 07Reception123 03137a223 - add ndg.nenawiki.org cert
[08:55:43] <Not-8d6a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJil
[08:55:44] <Not-8d6a>	 [02miraheze/ssl] 07Reception123 03b88fd70 - syntax fix
[15:37:58] <Not-8d6a>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[15:38:00] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/fhJHA
[15:38:01] <Not-8d6a>	 [02miraheze/puppet] 07paladox 031ac6392 - Add custom fact real_processor_count
[15:38:03] <Not-8d6a>	 [02puppet] 07paladox opened pull request 03#947: Add custom fact real_processor_count - 13https://git.io/fhJHx
[15:39:18] <Not-8d6a>	 [02puppet] 07paladox synchronize pull request 03#947: Add custom fact real_processor_count - 13https://git.io/fhJHx
[15:39:19] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/fhJHj
[15:39:21] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03635a517 - Update and rename real_processor_count.rb to virtual_processor_count.rb
[15:39:44] <Not-8d6a>	 [02puppet] 07paladox edited pull request 03#947: Add custom fact virtual_processor_count - 13https://git.io/fhJHx
[15:47:04] <Not-8d6a>	 [02puppet] 07paladox closed pull request 03#947: Add custom fact virtual_processor_count - 13https://git.io/fhJHx
[15:47:06] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fhJQm
[15:47:07] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03b7a331d - Add custom fact virtual_processor_count (#947) * Add custom fact real_processor_count * Update and rename real_processor_count.rb to virtual_processor_count.rb
[15:47:49] <Not-8d6a>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[15:47:51] <Not-8d6a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[15:48:42] <Not-8d6a>	 [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS
[15:48:44] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fhJQZ
[15:48:45] <Not-8d6a>	 [02miraheze/puppet] 07paladox 035870672 - Use new custom fact "virtual_processor_count" instead of ['processor']['count']
[15:48:47] <Not-8d6a>	 [02puppet] 07paladox opened pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor'… - 13https://git.io/fhJQn
[15:48:56] <Not-8d6a>	 [02puppet] 07paladox edited pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor']['count'] - 13https://git.io/fhJQn
[15:49:35] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fhJQl
[15:49:36] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03263ade6 - Update pool.pp
[15:49:37] <Not-8d6a>	 [02puppet] 07paladox synchronize pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor']['count'] - 13https://git.io/fhJQn
[15:51:31] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fhJQR
[15:51:33] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03eae6acc - Update nginx.conf.erb
[15:51:34] <Not-8d6a>	 [02puppet] 07paladox synchronize pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor']['count'] - 13https://git.io/fhJQn
[15:52:32] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fhJQE
[15:52:33] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03af06acb - Update nrpe.cfg.erb
[15:52:35] <Not-8d6a>	 [02puppet] 07paladox synchronize pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor']['count'] - 13https://git.io/fhJQn
[15:52:49] <Not-8d6a>	 [02puppet] 07paladox edited pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor']['count'] - 13https://git.io/fhJQn
[15:52:52] <Not-8d6a>	 [02puppet] 07paladox closed pull request 03#948: Use new custom fact "virtual_processor_count" instead of ['processor']['count'] - 13https://git.io/fhJQn
[15:52:54] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fhJQu
[15:52:55] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03d2d9354 - Use new custom fact "virtual_processor_count" instead of ['processor']['count'] (#948) * Use new custom fact "virtual_processor_count" instead of ['processor']['count'] * Update pool.pp * Update nginx.conf.erb * Update nrpe.cfg.erb
[15:52:57] <Not-8d6a>	 [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS
[15:52:58] <Not-8d6a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3
[15:55:28] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJQo
[15:55:30] <Not-8d6a>	 [02miraheze/puppet] 07paladox 037adbab8 - Fix loading custom fact in nrpe.cfg.erb
[15:55:58] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJQ6
[15:56:00] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03d114b16 - Fix loading custom fact in nginx.conf.erb
[16:09:18] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJQA
[16:09:19] <Not-8d6a>	 [02miraheze/puppet] 07paladox 039885a97 - some symlinks for the puppet master
[16:11:08] <paladox>	 !log depool mw1
[16:11:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:15:40] <paladox>	 !log repool mw1
[16:15:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:16:40] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJ7Z
[16:16:41] <Not-8d6a>	 [02miraheze/puppet] 07paladox 039a4af69 - php: Add support for puppet6
[16:21:36] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJ7B
[16:21:37] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03f60a948 - Update ordered_yaml.rb
[16:21:53] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJ70
[16:21:54] <Not-8d6a>	 [02miraheze/puppet] 07paladox 034421c13 - Update php_ini.rb
[16:22:06] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJ7E
[16:22:07] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03392174f - Update ini.rb
[16:25:39] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJ7V
[16:25:41] <Not-8d6a>	 [02miraheze/puppet] 07paladox 039367da9 - Update puppet.pp
[16:49:11] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:51:50] <icinga-miraheze>	 PROBLEM - test1 php7.2-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:52:10] <icinga-miraheze>	 PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:52:14] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:52:24] <icinga-miraheze>	 PROBLEM - test1 SSH on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:52:36] <icinga-miraheze>	 PROBLEM - Host test1 is DOWN: PING CRITICAL - Packet loss = 100%
[16:54:35] <icinga-miraheze>	 RECOVERY - Host test1 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms
[16:54:35] <icinga-miraheze>	 PROBLEM - test1 Disk Space on test1 is CRITICAL: connect to address 185.52.2.243 port 5666: Connection refusedconnect to host 185.52.2.243 port 5666: Connection refused
[16:54:35] <paladox>	 !log rebooted test1 to clear ram
[16:54:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:55:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJ5D
[16:55:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 0396c70db - BOT: Updating services config for wikis
[17:21:02] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[17:21:26] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.75, 1.36, 0.93
[17:22:15] <icinga-miraheze>	 RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 15081 MB (36% inode=98%);
[17:23:23] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.23, 1.44, 1.02
[17:23:25] <icinga-miraheze>	 RECOVERY - test1 php7.2-fpm on test1 is OK: PROCS OK: 2 processes with command name 'php-fpm7.2'
[17:24:01] <icinga-miraheze>	 RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 22921 bytes in 0.021 second response time
[17:28:26] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJdM
[17:28:28] <Not-8d6a>	 [02miraheze/puppet] 07paladox 034c48208 - fix path
[17:35:39] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJFe
[17:35:40] <Not-8d6a>	 [02miraheze/puppet] 07paladox 033e40f5c - fix loading the templates
[17:45:13] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJFl
[17:45:14] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03fe58024 - fix require
[17:53:09] <icinga-miraheze>	 PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.71, 1.41, 1.05
[17:55:05] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJFM
[17:55:06] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0370d2ba7 - vmlib: Add support for puppet6 in require_package
[17:55:09] <icinga-miraheze>	 RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.30, 0.95, 0.92
[18:12:00] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJbC
[18:12:01] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0399d9e1a - puppetmaster: symnlink /etc/puppet/private to /etc/puppetlabs/puppet/private
[18:17:31] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJbz
[18:17:33] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03cb9f3e9 - make test1 a temp puppetmaster
[18:27:42] <icinga-miraheze>	 PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[18:30:45] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJb5
[18:30:46] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03046f737 - Update site.pp
[18:33:27] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJbx
[18:33:28] <Not-8d6a>	 [02miraheze/puppet] 07paladox 03ff6e5d8 - Update test1.yaml
[18:36:21] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJNk
[18:36:23] <Not-8d6a>	 [02miraheze/puppet] 07paladox 035176f59 - Update puppet.pp
[18:37:42] <icinga-miraheze>	 RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 22921 bytes in 0.009 second response time
[18:55:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJN9
[18:55:13] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 039f78240 - BOT: Updating services config for wikis
[19:57:48] <Cyberpower678>	 paladox: JohnLewis: I'm still suffering from firewall issues when my production installation tries to communicate with the DB
[19:59:00] <Cyberpower678>	 DB Error: Unable to connect to the Wikipedia DB. Using failover. Please report if this message persists.
[19:59:14] <paladox>	 ah, ok. What's it's public ip?
[19:59:35] <Cyberpower678>	 Whatever the webservice for tools.wmflabs.org comes from.
[19:59:44] <Cyberpower678>	 Weren't we going to use a domain resolver?
[20:00:08] <paladox>	 yeh but we carn't do domain resolver
[20:00:51] <paladox>	 it dosen't seem to work for us
[20:00:55] <paladox>	 with gdnsd
[20:00:59] <paladox>	 *ufw
[20:03:48] <Cyberpower678>	 paladox: ping resolves to 208.80.155.131
[20:04:03] <paladox>	 i have that in https://github.com/miraheze/puppet/blob/adfe3ae788a4b74ed27752df09e3458a0135d804/modules/role/manifests/db.pp#L104
[20:04:04] <ZppixBot>	 Title: [ puppet/db.pp at adfe3ae788a4b74ed27752df09e3458a0135d804 · miraheze/puppet · GitHub ] - github.com
[20:04:14] <paladox>	 though i wonder what the ip is for when it connects to services
[20:04:34] <Cyberpower678>	 Hmmm...
[20:04:52] <Cyberpower678>	 Let me see if I can get it to echo the connection error.
[20:05:10] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJxb
[20:05:11] <Not-8d6a>	 [02miraheze/services] 07MirahezeSSLBot 03edef618 - BOT: Updating services config for wikis
[20:06:18] <paladox>	 does ip addr work for you?
[20:07:01] <JohnLewis>	 if it helps... I see failures to 3306 from 208.80.155.255 which is registered to the WMF
[20:07:41] <paladox>	 ah thanks!
[20:08:01] <JohnLewis>	 https://github.com/wikimedia/operations-dns/blob/master/templates/155.80.208.in-addr.arpa#L211
[20:08:01] <ZppixBot>	 Title: [ operations-dns/155.80.208.in-addr.arpa at master · wikimedia/operations-dns · GitHub ] - github.com
[20:08:19] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhJpe
[20:08:20] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0312e972d - whitelist another cyber power ip
[20:09:29] <Cyberpower678>	 paladox: JohnLewis: Now connecting
[20:09:29] <JohnLewis>	 keep in mind that might just be ONE of many addresses
[20:09:34] <paladox>	 Cyberpower678 try now
[20:09:47] <Cyberpower678>	 paladox: It's connecting now
[20:09:53] <paladox>	 :)
[20:10:25] <Cyberpower678>	 :-)
[20:26:27] <paladox>	 !log depooling mw1
[20:26:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:31:39] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures
[20:32:22] <icinga-miraheze>	 PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures
[21:59:20] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[21:59:42] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3
[21:59:44] <icinga-miraheze>	 PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3803 bytes in 0.009 second response time
[22:00:59] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[22:01:39] <icinga-miraheze>	 RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23696 bytes in 0.016 second response time
[22:01:41] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[22:03:06] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[22:05:06] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 50%
[22:07:05] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[22:08:43] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[22:09:06] <icinga-miraheze>	 RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 28%
[22:26:49] <ZppixBot>	 Hello Dan_! If you have any questions feel free to ask and someone should answer soon.
[22:40:22] <icinga-miraheze>	 RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[22:47:39] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:11:36] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 59963 MB (16% inode=96%);