[07:00:25] 10serviceops, 10Operations, 10User-Joe: SRE FY2019 Q3 goal: Ramp-up serving traffic to PHP 7 - https://phabricator.wikimedia.org/T212828 (10Joe) [07:01:01] <_joe_> @all: please take a look ^^ [09:05:50] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10Core Platform Team Backlog (Watching / External), 10Services (watching): TEC3:O3:O3.1:Q3 Goal - Move cxserver, citoid, changeprop, eventgate (new service) and ORES (partially) through the production ... - https://phabricator.wikimedia.org/T212801 [12:47:26] _joe_: hi, are you around? [12:47:47] <_joe_> leszek_wmde: for a few more minutes, yes :) [12:47:57] _joe_: I'll try to be fast then :) [12:48:52] _joe_: I was talking about T212189 with addshore and some other WMDE folks, and we figured it might be useful that we had a quick chat at some point, to explain more precisely what we're trying to achieve, and discuss some questions and concerns [12:49:37] _joe_: we're going to write down the RFC ticket nonetheless, but it seems that having more interactive conversation would also help to identigy what we need to define more etc [12:50:03] <_joe_> sure! [12:50:06] _joe_: would you have ~60 mins of spare time to meet over some video conference tool to have a little chat? [12:50:17] <_joe_> let's schedule a meeting early next week [12:50:23] _joe_: cool [12:50:31] <_joe_> so that it happens before the techcom meeting on wednesday [12:50:33] _joe_: you're in CET, aren't you? [12:50:36] <_joe_> yes [12:50:50] <_joe_> and I'm a morning person, anything after 8 am is ok :D [12:50:56] _joe_: would there be any time on Monday that would work for you? [12:51:11] alright [12:51:41] how about monday 11 am? [12:51:50] <_joe_> any time between 11 am - 1 pm is a good time [12:52:15] _joe_: brilliant! is google meet fine for you? [12:52:32] <_joe_> yes [12:52:53] <_joe_> I might add more people from my team to the meeting, if they're available [12:53:01] _joe_: fantastic. Invite will follow shortly. Thanks and we're happy to talk with you [12:53:09] _joe_: that'd be great, thanks! [12:53:14] <_joe_> likewise :) [14:07:46] do we meet this afternoon? Mark is out (see email), Alex is out (vacation), and we are just coming off of the holidays. But maybe there are enough of us here with things we want to discuss? [14:08:34] I am ok with skipping the meeting, we don't have much to say really [14:09:02] let's wait for jo.e [14:10:53] I have no strong feelings either way, but I expect thcipriani to wake up in a couple of hours and say something [14:22:19] liw: would y'all be making guest appareances today? [14:24:15] I think he means the pipeline meeting afterwards [14:25:28] er, yes, the pipeline meeting (which thcipriani has just cancelled) [14:28:20] * liw is a brain with very little bear [14:28:24] ah. no I just menanf the subteam meeting (the subteam for this channel, heh) [14:28:27] *meant [14:29:14] let's see what jo e and others say when they are around [14:42:18] <_joe_> I don't mind either way [14:44:24] <_joe_> but yeah unless fsero and jijiki want to meet to discuss things to do, I'm ok with skipping [14:47:36] James_F: are you around? [14:47:50] Yup. [14:48:09] How can I help? [14:48:18] I'll merge 481796 [14:48:23] i can always discuss things with you but if you feel like skipping it's ok too [14:48:46] jijiki: Does 481795 need to land first? [14:50:42] oh sorry [14:50:49] I meant the DNS one first :) [14:51:00] OK, good. :-) [14:51:16] <_joe_> it's not strictly needed, though [14:51:48] <_joe_> but anyways, let's roll with the dns change first [15:09:50] jijiki: DNS looks good to me, thanks! [15:10:33] <_joe_> jijiki: I'd go on with the apache change now [15:10:52] yeah [15:11:12] I was checking James_F's mediawiki config patch [15:11:18] the plot thickend :p [15:11:38] The MW config patch is probably not ready. [15:11:40] <_joe_> jijiki: welcome to our too complex recipe to add a new wiki [15:11:54] I'm sort-of hoping that a wild Reedy will appear and give me reassurance. ;-) [15:12:00] But yeah, MW config is… complex. [15:16:46] imagine adding a wiki and it only takes 4 minutes [15:17:20] (dc switch downtime is that short, so... 2019 goals :-D) [15:18:12] Yeah, well. [15:18:26] Static config for wikis (and massively reduced config surface) would help a lot. [15:18:58] I suppose there's a ticket (EPIC) for that... [15:20:15] Ha, [15:20:17] Docker first. [15:23:04] :-) [15:23:39] <_joe_> btw, I think we should meet [15:23:45] <_joe_> we can keep the meeting short [15:23:48] heh [15:23:55] ok, this means there are $things to discuss [15:24:45] <_joe_> well it's the beginning of the quarter [15:24:54] <_joe_> we might try to think ahead a bit :) [15:25:10] <_joe_> James_F: oh regarding mwconfig and dockerization... [15:25:18] <_joe_> we will have to work on it [15:25:35] Yeah. [15:26:11] Right now we have a single config set-up for all flavours; I imagine we're going to have to move to having version-specific config in some ways. It's going to be messy. [15:29:26] does anyone knows why this was done https://gerrit.wikimedia.org/r/c/operations/puppet/+/481132 /cc _joe_ [15:29:43] akosiaris: (who is probably on holidays) [15:30:14] <_joe_> fsero: I guess to restrict access? [15:30:15] while helping moritz to rollout new security upgrades i've found out some troubles with that [15:30:23] <_joe_> ok [15:30:30] i guess, but is also blocking calico to set up new ips per pods [15:30:33] <_joe_> we can make that a bit less stringent [15:30:38] <_joe_> ok [15:30:40] because cni needs to talk to etcd too [15:30:42] <_joe_> let's revert then [15:30:48] ack [15:30:53] <_joe_> or, fix it [15:30:58] or maybe we can expand to include the workers [15:31:19] <_joe_> we need the kube nodes to talk to etcd? [15:31:21] <_joe_> heh [15:31:26] calico needs it [15:31:31] :⁾ [15:31:40] <_joe_> hence why I wanted to have separated etcd clusters [15:31:42] <_joe_> anyways [15:31:57] <_joe_> ok it's not easy to get right [15:32:01] <_joe_> just revert for now [15:32:05] ack [15:32:31] we can also add them to the ferm service, it's just a total of eight nodes [15:32:38] but reverting also works for me [15:33:40] <_joe_> moritzm: and what about when we add new ones? [15:33:53] <_joe_> it needs to be automatic [15:33:53] lets revert it and fill a task to redoit in another way [15:33:58] <_joe_> yes [15:34:14] <_joe_> the best way would be to upgrade to etcd 3 and use client cert auth [15:41:33] revert and merged [15:43:03] we already have some existing Hiera entries where all k8s workers are listed (like for the bgp_peers), but let's sort this out a different time [15:43:58] James_F: looks ok [15:44:19] jijiki: Yeah, I think so too. Thank you! [15:44:27] the change will propagate slowly but you can test on mwdebug1001 if you like [15:44:31] :) [15:52:36] fsero: _joe_ apergos so we are on at :30 ? [15:52:50] <_joe_> I would say so [15:52:55] <_joe_> let's keep it short ofc [15:53:06] ok [15:53:11] <_joe_> have you all taken a look at our quarterly goals? [15:53:33] not the final final versions [15:53:38] yes we are on [15:53:38] <_joe_> I'll create an etherpad [16:16:39] <_joe_> everyone, the etherpad is https://etherpad.wikimedia.org/p/SRE-ServiceOps-2019-01-03 [16:43:45] 10serviceops, 10Operations, 10TechCom-RFC, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10WMDE-leszek) Thanks everyone for comments so far. This ticket in its current state is definitely not a ready RFC, you're right. We're going to turn i... [17:17:35] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Operations: Find an alternative to HHVM curl connection pooling for PHP 7 - https://phabricator.wikimedia.org/T210717 (10EBernhardson) >>! In T210717#4827144, @Joe wrote: > I see another problem here: > > say we do what makes sense and make MediaWiki con... [17:20:54] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Operations: Find an alternative to HHVM curl connection pooling for PHP 7 - https://phabricator.wikimedia.org/T210717 (10EBernhardson) I suppose more generally, the previous would be cleaner if we had some way to "tag" servers and check those server tags... [18:38:55] would this be a #serviceops task? https://phabricator.wikimedia.org/T203625#4779223 [20:23:06] 10serviceops, 10Operations, 10User-Joe: SRE FY2019 Q3 goal: Ramp-up serving traffic to PHP 7 - https://phabricator.wikimedia.org/T212828 (10herron) p:05Triage→03Normal