[00:02:53] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:03:39] <icinga-miraheze>	 PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues  
[00:10:49] <MacFan4000>	 !log (for transparency) using pywikibot to delete broken redirects on nonsensopediawiki per request
[00:10:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[00:12:54] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:13:38] <icinga-miraheze>	 RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[00:37:30] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpRqE
[00:37:31] <Not-2d3>	 [02miraheze/mw-config] 07paladox 031e9208e - Use local file storage for cdb files
[00:37:41] <paladox>	 SPF|Cloud ^^
[00:37:53] <SPF|Cloud>	 you're awesome
[01:25:10] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpRmg
[01:25:12] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 03d40bf02 - BOT: Updating services config for wikis
[01:39:11] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpRmQ
[01:39:12] <Not-2d3>	 [02miraheze/puppet] 07paladox 037b3776a - Fix vi /etc/icingaweb2/roles.ini to support latest icingaweb2
[01:39:32] <paladox>	 !log upgraded icingaweb2 to 2.6.2-1 on misc1
[01:39:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[01:51:23] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpRYU
[01:51:24] <Not-2d3>	 [02miraheze/puppet] 07paladox 03da4df55 - Increase MaxConcurrentChecks to 2
[16:42:25] <Not-2d3>	 [02mw-config] 07CnocBride opened pull request 03#2562: Upping SVGMetadataCutoff for magnaverse - 13https://git.io/fpRb3
[16:44:19] <Not-2d3>	 [02mw-config] 07paladox closed pull request 03#2562: Upping SVGMetadataCutoff for magnaverse - 13https://git.io/fpRb3
[16:44:21] <Not-2d3>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpRbn
[16:44:22] <Not-2d3>	 [02miraheze/mw-config] 07CnocBride 034fa1bf7 - Upping SVGMetadataCutoff for magnaverse (#2562) Getting issues with SVG files. Same error as T3278, this hopefully will resolve the issue.
[18:05:11] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpRxH
[18:05:13] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 0356b267f - BOT: Updating services config for wikis
[19:34:26] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+7/-0/±2] 13https://git.io/fp0e1
[19:34:28] <Not-2d3>	 [02miraheze/puppet] 07paladox 0385ede2d - Inital import of letsencrypt
[19:36:10] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp0eQ
[19:36:11] <Not-2d3>	 [02miraheze/puppet] 07paladox 03ae8bee1 - Update ssl-acme
[19:38:56] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp0eA
[19:38:57] <Not-2d3>	 [02miraheze/puppet] 07paladox 030048fe8 - Quiet certbot script Does not need to show any output.
[19:50:13] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp0vR
[19:50:14] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 03abe07b6 - BOT: Updating services config for wikis
[19:54:49] <JohnLewis>	 paladox: we shouldn't be catting the private key
[20:08:38] <paladox>	 JohnLewis see acme
[20:08:43] <paladox>	 though ah
[20:08:43] <paladox>	 ok
[20:10:08] <JohnLewis>	 well new security practises then okay :P
[20:10:14] <JohnLewis>	 except for LE stuff
[20:10:24] <JohnLewis>	 we shouldn't cat it for just CSRs
[20:11:19] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0fn
[20:11:21] <Not-2d3>	 [02miraheze/puppet] 07paladox 03361424a - Stop showing private ssl key when generating a cert
[20:11:22] <Not-2d3>	 [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS
[20:11:24] <Not-2d3>	 [02puppet] 07paladox opened pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[20:11:43] <paladox>	 JohnLewis :D
[20:12:15] <paladox>	 I want to convert that script to python at some point using the argument class
[20:12:17] <paladox>	 JohnLewis ^
[20:12:43] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0fl
[20:12:44] <Not-2d3>	 [02miraheze/puppet] 07paladox 039993c06 - Update ssl-acme
[20:12:46] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[20:13:40] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0f8
[20:13:42] <Not-2d3>	 [02miraheze/puppet] 07paladox 030c2a43d - Update ssl-certificate
[20:13:43] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[20:16:04] <paladox>	 JohnLewis i hope we will never have to use -g <domain> -s <domain> -s <domain (notice -s twice)
[20:18:04] <JohnLewis>	 We won’t 
[20:18:27] <JohnLewis>	 because -s is intentionally “second” domain not “extra” ;)
[20:20:46] <paladox>	 ok
[20:20:57] <paladox>	 JohnLewis we could do -d *.test.org
[20:21:03] <paladox>	 *-g
[21:14:31] <MacFan4000>	 Currently we show the privkey when we generate certain with acme-tiny
[21:14:53] <MacFan4000>	 s/certain/certs
[21:14:53] <ZppixBot>	 MacFan4000 meant to say: Currently we show the privkey when we generate certs with acme-tiny
[21:15:35] <paladox>	 that's why he said
[21:15:35] <paladox>	 <+JohnLewis>	well new security practises then okay :P
[21:15:37] <paladox>	 after :P
[22:00:16] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp0kK
[22:00:17] <Not-2d3>	 [02miraheze/services] 07MirahezeSSLBot 03f9392af - BOT: Updating services config for wikis
[22:07:21] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0kd
[22:07:22] <Not-2d3>	 [02miraheze/puppet] 07paladox 03b078cdb - Update ssl-certificate
[22:07:24] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[22:07:43] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0kx
[22:07:44] <Not-2d3>	 [02miraheze/puppet] 07paladox 03d34473a - Update ssl-acme
[22:07:46] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[22:08:00] <Not-2d3>	 [02puppet] 07paladox closed pull request 03#898: Redirect /.well-known/acme-challenge/ to mw1.miraheze.org - 13https://git.io/fp45v
[22:08:03] <Not-2d3>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2
[22:08:05] <Not-2d3>	 [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS
[22:08:21] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[22:08:22] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0kh
[22:08:24] <Not-2d3>	 [02miraheze/puppet] 07paladox 03dd0a3cf - Update ssl-acme
[22:33:06] <JohnLewis>	 !log depool mw2
[22:33:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:35:17] <Not-2d3>	 [02puppet] 07paladox synchronize pull request 03#899: Stop showing private ssl key when generating a cert - 13https://git.io/fp0fc
[22:35:18] <Not-2d3>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fp0Ip
[22:35:20] <Not-2d3>	 [02miraheze/puppet] 07paladox 03429369f - Support wildcards
[22:38:51] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down.  mw2  
[22:39:28] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down.  mw2  
[22:39:52] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down.  mw2  
[22:51:37] <paladox>	 hmm
[22:59:05] <paladox>	 nvm
[22:59:08] <paladox>	 see the log above
[23:02:55] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy  
[23:03:29] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy  
[23:03:53] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy  
[23:24:27] <revi>	 Canvassing: https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2019/Anti-harassment/Add_an_option_to_require_email_address_and_username_to_reset_password >_<
[23:24:27] <ZppixBot>	 Title: [ Community Wishlist Survey 2019/Anti-harassment/Add an option to require email address and username to reset password - Meta ] - meta.wikimedia.org
[23:24:43] <revi>	 will also work on Miraheze anyway if they do it
[23:36:07] <icinga-miraheze>	 PROBLEM - cp2 SSH on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[23:36:11] <icinga-miraheze>	 PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[23:36:13] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb  
[23:36:21] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[23:36:41] <icinga-miraheze>	 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.  
[23:37:09] <icinga-miraheze>	 PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3801 bytes in 0.403 second response time   
[23:37:57] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb  
[23:38:13] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online  
[23:39:00] <paladox>	 uh
[23:39:18] <paladox>	 JohnLewis are you aware of ^^?
[23:39:46] <paladox>	 ssh is not working for me
[23:40:17] <icinga-miraheze>	 RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures  
[23:40:38] <paladox>	 works now
[23:41:09] <icinga-miraheze>	 RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 23696 bytes in 0.600 second response time   
[23:41:59] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online  
[23:42:15] <icinga-miraheze>	 RECOVERY - cp2 SSH on cp2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u4 (protocol 2.0)   
[23:42:21] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy  
[23:42:47] <icinga-miraheze>	 RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2%  
[23:45:21] <Not-2d3>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp0q9
[23:45:22] <Not-2d3>	 [02miraheze/dns] 07paladox 03c96a6c3 - depool cp2
[23:45:38] <paladox>	 !log depooling cp2 for upgrade to debian 9.6
[23:45:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:52:00] <Not-2d3>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp0qx
[23:52:01] <Not-2d3>	 [02miraheze/dns] 07paladox 033f3cd6e - Revert "depool cp2" This reverts commit c96a6c3a165651e1e61dbf862ad606760ebf716c.
[23:53:00] <paladox>	 !log repool cp2
[23:53:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master