[10:29:52] <arturo>	 !log admin T219626 replace labtestcontrol2003 with cloudcontrol2001-dev in the clouddb2001-dev database (codfw1dev deployment)
[10:30:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:30:16] <stashbot>	 T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch - https://phabricator.wikimedia.org/T219626
[10:37:00] <arturo>	 !log admin T219626 replace 208.80.153.75 with 208.80.153.59 in the clouddb2001-dev database (codfw1dev deployment)
[10:37:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:37:03] <stashbot>	 T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch - https://phabricator.wikimedia.org/T219626
[11:27:42] <arturo>	 !log admin T219626 add DB grants for neutron and glnace to clouddb2001-dev (codfw1dev)
[11:27:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:27:45] <stashbot>	 T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch - https://phabricator.wikimedia.org/T219626
[12:07:57] <arturo>	 !log admin rebooting cloudvirt200[123]-dev because deep changes in config
[12:08:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[17:00:56] <andrewbogott>	 !log tools moving tools-k8s-master-01 to eqiad1-r
[17:00:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:07:01] <chicocvenancio>	 !log paws move paws-proxy-02 to point to tools-paws-worker-1006 for upcoming master move
[17:07:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[17:15:01] <chicocvenancio>	 !log tools add paws outage announcement in configmap hub-config
[17:15:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:15:25] <chicocvenancio>	 !log paws move paws-proxy-02 reload nginx 
[17:15:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[17:30:55] <andrewbogott>	 !log tools moving tools-paws-master-01 to eqiad1-r
[17:35:17] <bd808>	 !log tools.stashbot Test
[17:35:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[20:22:37] <dbrant>	 o/  Is this the right place to ask someone about SSL certificates on Beta Cluster machines?
[20:23:36] <andrewbogott>	 dbrant: here, or in -releng
[20:24:05] <andrewbogott>	 The answer to most deployment-prep questions is: ask Krenair or Tyler
[20:24:15] <andrewbogott>	 and Tyler (aka thcipriani) is usually in -releng
[20:24:26] <Krenair>	 hi dbrant 
[20:26:10] <dbrant>	 heya
[20:26:19] <dbrant>	 So here's the issue:  when trying to connect to certain beta servers from the Android app, we've started getting certificate validation errors, along the lines of not being able to determine revocation status. (even though the cert looks ok when browsing through Chrome)...
[20:26:55] <Krenair>	 interesting
[20:27:08] <dbrant>	 I'm not an expert in this stuff, but the only lead I've been able to track is the following: https://github.com/DmcSDK/cordova-plugin-mediaPicker/issues/69#issuecomment-469666841
[20:27:30] <dbrant>	 "OCSP Stapling renewal"
[20:28:02] <Krenair>	 well the first thing to know is the host doing TLS termination for beta, with the exception of upload, is deployment-cache-text05
[20:28:20] <Krenair>	 as with prod it's done by nginx and the config is at /etc/nginx/sites-enabled/unified
[20:28:42] <Krenair>	 which does contain
[20:28:42] <Krenair>	 	ssl_stapling on;
[20:28:43] <Krenair>	 	ssl_stapling_file /etc/acmecerts/unified/live/rsa-2048.client.ocsp;
[20:28:43] <Krenair>	 	ssl_stapling_file /etc/acmecerts/unified/live/ec-prime256v1.client.ocsp;
[20:32:17] <Krenair>	 I'm not very familiar with how OCSP works
[20:35:49] <Krenair>	 well
[20:36:02] <dbrant>	 :/  nor am I, but it seems like the likely culprit.
[20:36:10] <Krenair>	 root@deployment-cache-text05:~# openssl ocsp -issuer /etc/acmecerts/unified/live/rsa-2048.chain.crt -cert /etc/acmecerts/unified/live/rsa-2048.crt -text -url http://ocsp.int-x3.letsencrypt.org
[20:36:10] <Krenair>	 ...
[20:36:22] <Krenair>	 Response verify OK
[20:36:23] <Krenair>	 	This Update: Apr 16 09:00:00 2019 GMT
[20:36:23] <Krenair>	 	Next Update: Apr 23 09:00:00 2019 GMT
[20:36:36] <Krenair>	 same for the ECDSA cert
[20:38:00] <Krenair>	 dbrant, you don't have anything hardcoded about checking revocation status against DigiCert/GlobalSign but not LetsEncrypt do you?
[20:38:13] <dbrant>	 no, nothing like that
[20:38:52] <Krenair>	 can you show the error message you're getting?
[20:39:31] <Krenair>	 IIRC Firefox cares a lot more about OCSP than Chrome but the site is working for me in FF
[20:41:15] <dbrant>	 it's basically the same stack trace as the github issue I mentioned:  https://github.com/DmcSDK/cordova-plugin-mediaPicker/issues/69#issuecomment-469597770
[20:42:30] <Krenair>	 dbrant, is the clock on this device accurate?
[20:42:53] <dbrant>	 yes, it happens on any device.
[20:43:29] <Krenair>	 We did have a user experience OCSP verification issues on prod recently and it turned out to be because their clock was wildly wrong, plus this stack trace has an exception about 'Response is unreliable: its validity interval is out-of-date'
[20:43:33] <Krenair>	 ok
[20:45:03] <Krenair>	 did this problem just start occurring recently?
[20:46:27] <dbrant>	 roughly within the last two or three weeks, but may be a bit longer.
[20:47:21] <Krenair>	 well
[20:47:26] <Krenair>	 I did redo how this all works recently
[20:49:02] <Krenair>	 root@deployment-cache-text05:~# /usr/lib/nagios/plugins/check-fresh-files-in-dir.py -c 259500 -w 173100 -d /var/cache/ocsp -g "*.ocsp"
[20:49:02] <Krenair>	 OK
[20:49:46] <Krenair>	 that seems to be what prod would do to monitor this stuff
[20:49:46] <chicocvenancio>	 !log tools change paws announcement in configmap hub-config back to a welcome message
[20:49:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:52:35] <Krenair>	 dbrant, is there any chance you could take the code you use to talk to wikis and test just the HTTP part against something in prod using LE certs?
[20:53:07] <Krenair>	 like https://librenms.wikimedia.org/ ? no need to log in, just curious about whether OCSP works there
[20:53:41] <dbrant>	 will do...
[20:54:04] <Krenair>	 that should help make it clear whether this is something specifically broken in deployment-prep or more generally about the new LE puppetisation
[20:54:10] <Krenair>	 is there a task for this?
[20:56:52] <dbrant>	 No task yet, just started to dig into it.
[20:57:11] <Krenair>	 ok
[20:57:24] <dbrant>	 OK, the request to the librenms site was quite successful.
[20:57:32] <Krenair>	 huh
[20:57:51] <Krenair>	 do you have something minimal to reproduce the problem with against beta?
[20:59:59] <dbrant>	 Well it would need to be using the Android app, and monitoring the logs coming from it.
[21:00:27] <Krenair>	 can we take the HTTP library you're using a make a tiny java program to dig into?
[21:00:35] <Krenair>	 the issue
[21:00:56] <Krenair>	 or does it rely on android APIs?
[21:02:00] <dbrant>	 I believe it's independent of Android... Let me try to slap something together.
[21:08:03] <Krenair>	 it'd be nice to be able to dig into exactly what it's expecting vs. what it gets from us
[21:08:21] <Krenair>	 and then figure out what is wrong from there
[21:08:35] <Krenair>	 plus it'd be nice to be able to reproduce it on an ordinary system instead of android
[21:24:48] <dbrant>	 Krenair: Here we go -- https://github.com/dbrant/okhttpssltest
[21:27:04] <Krenair>	 lets see if I can figure out how to build this
[21:27:50] <dbrant>	 oh sorry, you should just be able to check out the repo and run "./gradlew build" and then "./gradlew run"
[21:29:06] <bd808>	 after installing gradle ;)
[21:30:48] <Krenair>	 > Could not find tools.jar. Please check that /usr/local/java/jre1.8.0_112 contains a valid JDK installation.
[21:30:57] <Krenair>	 alex@alex-laptop:~/Development/Wikimedia/okhttpssltest (master)$ /usr/local/java/jre1.8.0_112/bin/java -version
[21:30:57] <Krenair>	 java version "1.8.0_112"
[21:31:43] <dbrant>	 that looks like a JRE, but do you have a JDK?
[21:32:34] <Krenair>	 no tools.jar or javac in there
[21:32:36] <Krenair>	 so am guessing no
[21:32:54] <Krenair>	 must have one somewhere as javac is in my path
[21:33:12] <Krenair>	 lrwxrwxrwx 1 root root 23 Feb  9  2017 /usr/bin/javac -> /etc/alternatives/javac
[21:33:18] <Krenair>	 lrwxrwxrwx 1 root root 43 Feb  9  2017 /etc/alternatives/javac -> /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
[21:33:24] <Krenair>	 maybe I can make it use that install
[21:34:55] <Krenair>	 not immediately obvious how
[21:36:25] <Krenair>	 there we go
[21:36:38] <Krenair>	 made a gradle.properties file with org.gradle.java.home=/usr/lib/jvm/java-8-openjdk-amd64
[21:37:55] <Krenair>	 alright so I built it and ran './gradlew run'
[21:38:13] <Krenair>	 > Task :run 
[21:38:13] <Krenair>	 Response successful!
[21:38:13] <Krenair>	 BUILD SUCCESSFUL in 1s
[21:38:13] <Krenair>	 2 actionable tasks: 1 executed, 1 up-to-date
[21:43:27] <Krenair>	 dbrant, was that supposed to fail?
[22:32:47] <dbrant>	 Krenair: sorry, stepped away.  Yes, it's supposed to fail. (it's failing for me)
[22:33:01] <Krenair>	 very interesting
[22:33:11] <Krenair>	 can you show the full output you get?
[22:34:26] <dbrant>	 https://pastebin.com/G8gTmRaX
[22:38:24] <dbrant>	 Curious -- when I run it from my MacBook, it works.
[22:49:49] <dbrant>	 it fails on my Windows desktop (and Android) but not Linux or macOS.
[22:55:19] <bd808>	 !log admin cloudcontrol2003-dev: added `exit 0` to /etc/cron.hourly/keystone to stop cron spam on partially configured cluster
[22:55:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[23:03:51] <Krenair>	 dbrant, interesting
[23:04:01] <Krenair>	 this does not have anything about OCSP
[23:05:54] <bd808>	 sounds like a root cert issue
[23:06:36] <Krenair>	 yes
[23:06:48] <Krenair>	 the thing is we do send the intermediate cert for browsers which do not trust LE
[23:06:56] <Krenair>	  0 s:/CN=*.wikimedia.beta.wmflabs.org
[23:06:57] <Krenair>	    i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
[23:07:03] <Krenair>	  1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
[23:07:03] <Krenair>	    i:/O=Digital Signature Trust Co./CN=DST Root CA X3
[23:16:54] <Krenair>	 I wonder if you're getting caught out by https://letsencrypt.org/certificates/#ocsp-signing-certificate dbrant 
[23:19:38] <Krenair>	 if the OCSP responses are signed by the ISRG root, how are they supposed to be used by browsers which rely on the DST root
[23:21:19] <Krenair>	 the other question is why would it work with librenms but not beta, their certs are issued by the same CA, the same intermediate is provided. . .
[23:28:46] <Krenair>	 I kind of have a feeling someone is -traffic is going to be able to take one look and know what is wrong
[23:30:20] <Krenair>	 dbrant, did you test it against librenms in windows?
[23:35:47] <dbrant>	 Hmm, on Windows librenms does *not* work, either.
[23:38:49] <bd808>	 https://letsencrypt.org/docs/certificate-compatibility/
[23:39:00] <bd808>	 "The main determining factor for whether a platform can validate Let’s Encrypt certificates is whether that platform includes IdenTrust’s DST Root X3 certificate in its trust store. A secondary factor is whether the platform supports modern SHA-2 certificates, since all Let’s Encrypt certificates use SHA-2."
[23:45:57] <dbrant>	 So, I'm curious what has changed between now and when it worked previously...