[09:00:13] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Build calico 3.16 - https://phabricator.wikimedia.org/T266893 (10Joe) As for the manifests, if we need them, they should be in hellfile.d/admin I guess? [09:12:48] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Build calico 3.16 - https://phabricator.wikimedia.org/T266893 (10JMeybohm) >>! In T266893#6605376, @Joe wrote: > As for the manifests, if we need them, they should be in hellfile.d/admin I guess? The thing with that is that it relies on other stuff to work. At... [12:29:48] Hi all, i have a CR out which updates the profile module to use the share spec_helper.rb. This means that the spec test use the real data from the hieradata folder as well as having access to the private repo and other dependent modules. This should hopefully make it much easier to write spec test for profiles in the future and drop the need to maintain additional hiera files in the [12:29:53] fixtures folder. I have ported all the current ... [12:29:56] ... tests to this new strutrue and change the tests to expect real values base on role and cluster but its definetly possible i have inverted somne test logic or accidently droped something which should be tested as such looking for reviwers [12:30:36] ping _joe_: as the main (almost only) commiter to modules/profile/spec/fixtures/hieradata [12:31:04] and a link to the CR https://gerrit.wikimedia.org/r/c/operations/puppet/+/638678/ [12:32:20] <_joe_> jbond42: uhm this means that some tests can't be run as we don't cover all the test cases with our hieradata though [12:32:32] <_joe_> I'll take a look soon though :) [12:33:22] _joe_: i think i have managed to find real data for all current tests but yes that is something lacking [12:33:29] and thx :) [12:33:51] <_joe_> don't get me wrong, this is a great thing (also I guess possible now with hiera 5) [12:34:22] <_joe_> and kudos for doing it [12:35:17] could have been possible with hiera3 but would have been super hacky if even dooable. the shared spec helper definetly existsed with hiera3 but i rember abandoning the PS to port profile to it so definetly easier now :) [12:36:26] <_joe_> one small issue I see with this is we'll need to change tests when stuff changes in prod like e.g. a hiera value [12:36:37] there are still some hacks in place currently mostly that globals are mocked as facts [12:37:17] <_joe_> I saw, probably it would be ok to mock them as node_params [12:37:20] <_joe_> which is less hacky [12:37:48] yes i need to dig into how to do that at the spec_helper level but i agree [12:38:03] <_joe_> but I didn't look at how to do it in a spec_helper, yes [12:40:12] in relation to data changing in production i'd say lets wait and see how much of an issue it is. i dont think many tests are that specific. in fact the mediawiki ones testing the lvs ips are probably the most specific ones [13:02:40] <_joe_> ack :) [13:26:45] hi all looking at a change related to confd (https://gerrit.wikimedia.org/r/c/operations/puppet/+/617716) and notice that the current config has `confd::srv_dns: 'eqiad.wmnet'` in all sites except esams where it is `confd::srv_dns: 'esams.wmnet'` is this correct? [13:30:00] ahh looks like this was part of the switch over work https://gerrit.wikimedia.org/r/c/operations/puppet/+/623535 (gussing esams just got accidentily missed in the initial PS) rzl/joe should we switch theses back to the site specific domain? [14:26:42] huh, probably but I'm not sure why esams was left out in the first place [14:26:53] very plausibly it just got missed but I'd like to get _joe_ to confirm before we touch it [14:27:46] rzl: esams goes to eqiad anyway IIRC [14:27:55] the SRV records [14:28:05] ahh is that what it was? I'd buy that [14:28:23] in that case yeah we can just revert 617716 [14:29:14] I kind of want to document it in https://wikitech.wikimedia.org/wiki/Switch_Datacenter just so we don't lose track of it, although hopefully it'll be obsolete by the time anyone reads that page next [14:42:33] thanks rzl have created https://gerrit.wikimedia.org/r/c/operations/puppet/+/639492 [17:06:11] 10serviceops, 10DC-Ops, 10Operations, 10ops-eqiad: eqiad: Physical moves for MediaWiki servers - https://phabricator.wikimedia.org/T266164 (10Cmjohnson) mw1267 issues have been fixed [18:14:57] 10serviceops, 10Operations, 10ops-eqsin: ganeti5002 was down / powered off, machine check entries in SEL - https://phabricator.wikimedia.org/T261130 (10RobH) The reimage of this host is giving me trouble. I've verified in the idrac bios setttings that IPMI over lan is enabled, but the script errors out with... [18:35:49] where... where does the thankyou.wp.org apache config come from? [18:37:38] 10serviceops, 10Performance-Team, 10Platform Engineering Roadmap Decision Making, 10Code-Health-Objective, and 4 others: Determine multi-dc strategy for CentralAuth - https://phabricator.wikimedia.org/T267270 (10BPirkle) [18:51:13] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [18:51:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Build new kubernetes packages - https://phabricator.wikimedia.org/T266766 (10JMeybohm) 05Open→03Resolved 1.16.15 released to stretch-wikimedia component/kubernetes-future [18:55:15] cdanis: that's a really good question [18:55:37] traffic does send that domain to the appservers [18:55:40] I've verified that much so far [18:55:45] yeah same [18:56:32] so there's a default wikipedia server in apache and then there's a bunch of stuff about thankyouwiki in InitialiseSettings.php ? [18:56:35] is that all? [18:57:37] yeah I guess so! I expected to see more about it in the apache configs, but I guess that's just for redirects and other niceties that we don't need for thankyouwiki [18:58:03] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, and 2 others: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10AMooney) [18:58:03] since nobody uses it as an actual wiki [18:58:40] for the record it's also brand new, so it's *possible* whatever's weird about it is also just wrong and broken [18:58:55] wouldn't be my first guess, but I wouldn't take it for granted either [18:59:35] so we do need a custom docroot for it rzl [19:03:40] kk -- I don't have much context here but I can try and catch up [19:04:16] yeah sorry -- this is re: https://phabricator.wikimedia.org/T259312 [19:05:31] ahhhh okay [19:07:46] some relevant discussion rn in #-operations too [19:08:19] I suspect it would be enough to add another entry to mediawiki::web::vhost in prod_sites.pp? [19:10:18] that sounds right to me but I don't actually know my way around that config as well as I wish -- I'd like to get someone else's eyes on it too [19:10:42] normally I'd say m.utante but he's on vacation, it'll probably have to be j.oe tomorrow [19:10:52] yeah, sgtm [19:11:01] we can go ahead if it's urgent though, easy enough to try some stuff and test on mwdebug [19:11:24] tbh we can do that either way, disable puppet and futz with it on-host [19:11:29] bug opened Jul 30 so I don't think it is, I'll check with ejegg [19:11:48] either way though I need to eat some lunch first, oops 🙃 [19:11:53] oh no, please do [22:42:20] 10serviceops, 10Performance-Team, 10Platform Engineering Roadmap Decision Making, 10Code-Health-Objective, and 4 others: Determine multi-dc strategy for CentralAuth - https://phabricator.wikimedia.org/T267270 (10Krinkle) I think the task analysis here is accurate in the context of "main stash is moving to...