[05:29:58] jbond42: I was troubleshooting a reimage I am done today, signing the puppet new cert is failing due to: Error: Could not request certificate: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired):] [05:30:16] *I am doing today [05:30:25] Before that, I can see: Info: Not using expired certificate for ca from cache; expired at 2020-06-29 19:36:29 UTC [05:32:07] I did a destroy on puppetmaster for that specific host, but I still see: Info: Not using expired certificate for ca from cache; expired at 2020-06-29 19:36:29 UTC Error: Could not run: stack level too deep on the first puppet run on that host [05:33:48] Removing /var/lib/puppet/ssl looks like did the trick for that host [05:35:03] Ah no, it didn't puppet keeps failing on the first run with expired certificate [05:36:20] Hello ...I am also having a certificate error when pulling images from docker-registry.discovery.wmnet/wikimedia on kubernetes after deploying to staging with helm...I guess I will just roll back for now [06:13:44] I tried reverting and applying changes, but it just hung on pending_update, so I rolled back to the revision from before my initial deploy, which said it succeeded. For some reason there is a new pod still trying to pull the new image, though. [07:28:41] marostegui: what host, that is the exipry date of the old puppet CA [07:28:54] jbond42: db1080 [07:29:00] thx looking [07:29:06] I did destroy the old one, but looks like the new one is being created with the same date? [07:29:45] ack ill take a look [07:31:52] thank you [07:32:39] np [07:34:28] jbond42: i've filed https://phabricator.wikimedia.org/T256720 as discussed. next time it happens i'll update with whatever info i can get from network/js consoles in ff [07:36:06] kormat: thx [07:41:34] ¿¡we're moving off of gerrit?! [07:42:31] best birthday present i could have wished for :) [07:45:29] {{cn}} [07:47:15] https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/GitLab [07:53:58] marostegui: puppet is running now. Seems that an onld version of the CA was not replaced (/var/lib/puppet/server/ssl/ca/ca_crt.pem). this gets copied over as the CA.pem cert during install. once installed puppet replaces the ca.pem file with the correct certificate which is why the rest of the fleet is working. [07:54:27] I have manuly updated the ca_crt.pem file on the frontend puppetmasters and will update puppet to make sure this is automated [07:54:47] ill let you know when the first puppet run has finished [07:54:56] * jbond42 going to grab coffee now [07:56:15] thanks for fixing it :) [07:59:36] np hopfully this is the only issue missed :) (fyi https://phabricator.wikimedia.org/T256721) [08:44:32] jbond42: CI hates me because i added 2 new `hiera()` calls in a class which already uses `hiera()`. (https://gerrit.wikimedia.org/r/c/operations/puppet/+/608558). what's the expected approach? [08:44:52] mix new lookup()'s with old hiera()'s, change the existing hiera()'s to lookup()'s, something else? [08:48:41] kormat: so the way CI works is that it only alerts on new issues you introduce, so to make CI happy just change your hiera calls to lookup calls. to make me happy change all hiera calls to lookup calls and add types :) [08:48:57] haha, ok [08:52:55] thx and feel free to add me if you need a reviewer [08:54:41] i shall :) [09:39:01] there's also always "# lint:ignore:wmf_style" (which is sometimes perfectly adequate; given that CI only votes on changes in violations sometimes touching code which predates the style guide could otherwise lead into deep rabbit roles of refactoring=) [09:42:03] moritzm: ahh. that's good to know. ;) [15:15:44] is everyone getting this when opening gerrit since the v3 upgrade, or is it just me? https://usercontent.irccloud-cdn.com/file/OlQ38L86/image.png [15:16:10] rzl: you have to clear your browser cache [15:16:15] ah thanks [15:16:22] because web is shit. [15:18:59] * RhinosF1 hates browser cache [17:55:22] added FAQ section for issues after Gerrit upgrade to 3.2: https://wikitech.wikimedia.org/wiki/Gerrit#Troubleshooting_/_upgrade_FAQ [17:57:31] mutante: that should probably be on mw.org as most other user-facing documentation is there [17:59:35] Majavah: i disagree, mw.org is for MediaWiki. Wikitech is for docs of technical infrastructure. [18:00:51] mutante: Gerrit's user facing documentation is on mw.o and documentation related to maintaining the gerrit server is on wikitech [18:01:24] mw.o already has gerrit user documentation and all other documentation related to contributing to mediawiki software [18:01:53] wikitech has documentation related to maintaining Wikimedia deployment [18:03:55] I can copy it. I still see no relation to MediaWiki though. [18:04:20] dunno why it's being used for that [18:12:08] mediawiki.org has all of our user facing docs for phabricator, gerrit, etc [18:12:31] wikitech docs are really meant to be SRE facing (other than Toolforge/Cloud VPS) [18:13:57] I copied the text over to MediaWiki. Done, no big deal. I still don't agree though. "Wikitech is the home of documentation related to the technical projects and infrastructure maintained by the Wikimedia Foundation." [18:14:08] it's just as user-facing as other wikis [18:14:50] I'm moving them from [[Gerrit]] to [[Gerrit/Troubleshooting]] [18:14:51] it even says "including toolforge, cloud [18:16:26] you can dispute it, but the docs have been split this way for the 7 years I have worked here :) [18:17:14] wikitech having toolforge and VPS docs is a bug, not a feature honestly. Left over from the merge of labswiki and wikitechwiki before I got here [18:29:04] yea, i have been disputing it probably just as long