[10:49:36] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10PHP 7.2 support, 10User-jijiki: Improve Pybal's url checks - https://phabricator.wikimedia.org/T222705 (10jijiki)
[10:49:48] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10PHP 7.2 support, 10User-jijiki: Improve Pybal's url checks - https://phabricator.wikimedia.org/T222705 (10jijiki) p:05Triage→03High
[10:50:23] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10PHP 7.2 support, 10User-jijiki: Improve Pybal's url checks - https://phabricator.wikimedia.org/T222705 (10jijiki)
[10:50:26] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review, 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10jijiki)
[11:49:09] <wikibugs>	 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Build Thumbor packages for buster - https://phabricator.wikimedia.org/T221562 (10jijiki)
[12:04:45] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10akosiaris) >>! In T218346#5161770, @Ottomata wrote: > @akosiaris I think https...
[13:20:42] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Ok, sounds good.  I'll be deploying a new 'service' to the eventgate...
[13:25:08] <ottomata>	 o/
[13:25:40] <ottomata>	 i've got 2 main tasks as part of the eventgate chart refactor: 
[13:25:52] <ottomata>	 deploy eventgate-main, which we are discussiong on  ^^^ 
[13:25:58] <ottomata>	 discussing*
[13:26:07] <ottomata>	 and to replace the running eventgate-analytics chart
[13:26:25] <ottomata>	 Yesterday I deployed stable/eventgate -n analytics to staging
[13:26:32] <ottomata>	 I'd like to do that to eqiad and codfw today
[13:26:58] <ottomata>	 akosiaris:  you ok with that?  can I use the same resource requests & replicas as before?
[13:27:24] <ottomata>	 once it is up and good, is there anything other than changing the LVS backend port that would need to be done?
[13:27:52] <akosiaris>	 changing LVS ? why ?
[13:28:11] <ottomata>	 the replacement services uses a new port
[13:28:15] <ottomata>	 to not conflict with the old one
[13:28:20] <ottomata>	 service*
[13:28:29] <akosiaris>	 I think you can deploy the new chart on top of the old one
[13:28:37] <akosiaris>	 I 've tested it once and it worked IIRC
[13:28:40] <ottomata>	 with the same port?
[13:28:58] <akosiaris>	 on the same release, so yeah same port
[13:29:06] <ottomata>	 no its a new release name
[13:29:10] <akosiaris>	 that is if the 2 charts are functionally equivalent
[13:29:16] <ottomata>	 new chart, new release, same namespace/service
[13:29:51] <ottomata>	 e.g.
[13:29:53] <akosiaris>	 yeah I think you don't new the new release, just the new chart
[13:29:53] <ottomata>	 CLUSTER=staging scap-helm eventgate-analytics install -n analytics -f analytics/staging-values.yaml stable/eventgate
[13:30:06] <akosiaris>	 actually... I can just try that and make sure
[13:30:15] <ottomata>	 we need the new release beacuse it was the only way to keep the metric and other names the same.
[13:30:42] <akosiaris>	 aha, you no longer will be using production
[13:30:47] <ottomata>	 right.
[13:30:56] <ottomata>	 since there is one chart deploying multipel services
[13:31:13] <ottomata>	 https://phabricator.wikimedia.org/T218346#5144422
[13:31:40] <akosiaris>	 in that case it sounds fine to me
[13:32:06] <ottomata>	 cool, and we can use the same port?
[13:32:19] <akosiaris>	 but for the deployment of the new service (main IIRC?), I 'd rather sre-serviceops does the creation of the namespace and all
[13:32:20] <ottomata>	 or no, we need the new one not to conflict, right?
[13:32:39] <ottomata>	 ya main, that makes sense.
[13:32:41] <akosiaris>	 yeah you 'll need a new port
[13:32:51] <ottomata>	 ok, cool.  that's what I have on staging now.
[13:32:53] <akosiaris>	 and will have to update the lvs configuration to point to the new port
[13:33:07] <ottomata>	 ok great.  so i'll deploy new chart analytics to codfw and eqiad, same requests
[13:33:12] <ottomata>	 once it looks just asa good as the old one
[13:33:17] <ottomata>	 we can change over the LVS stuff
[13:33:21] <ottomata>	 and then take down the old one.
[13:33:26] <akosiaris>	 ok
[13:33:39] <ottomata>	 i'll do that now
[13:33:59] <ottomata>	 as for -main stuff, you'd rather yall just handled all patches too?  
[13:34:23] <ottomata>	 as in: "ottomata, hold your  horses we'll do it!"
[13:34:28] <akosiaris>	 oh I can't stop you from pushing patches :-). Just lets us review and merge them 
[13:34:33] <ottomata>	 heheh ok great.
[13:34:55] <ottomata>	 i don't know of  other things to patch other than https://gerrit.wikimedia.org/r/c/operations/puppet/+/508371
[13:35:07] <ottomata>	 (and tokens in private?) 
[13:37:41] <akosiaris>	 it's the LVS service as well, discovery records and LVS IPs on the kubernetes workers
[13:37:52] <ottomata>	 oh yes
[13:37:54] <akosiaris>	 oh and LVS IP assignment in DNS
[13:38:00] <ottomata>	 but i mean stuff to do before LVS
[13:38:09] <ottomata>	 we can deploy k8s service before LVS ya?
[13:38:13] <akosiaris>	 yup
[13:38:17] <akosiaris>	 there is also the calico stuff
[13:38:27] <akosiaris>	 modules/calico/data/default-kubernetes-policy.yaml:
[13:38:41] <akosiaris>	 you 'll see src_selector: calico/k8s_ns == 'eventgate-analytics' lines
[13:38:54] <akosiaris>	 I am guessing you need something similar but eventgate-main ?
[13:39:04] <akosiaris>	 also probably different IPs, right? different kafka cluster?
[13:39:14] <ottomata>	 yup
[13:39:31] <ottomata>	 Ah ha ok I see it
[13:39:35] <ottomata>	 will add to that patach
[13:40:13] <akosiaris>	 we will be getting rid of that this quarter hopefully
[13:40:16] <ottomata>	 oh ya?
[13:40:17] <ottomata>	 how?
[13:40:19] <akosiaris>	 it's a couple of upgrades that are pending 
[13:40:51] <akosiaris>	 we 'll be moving that to the /srv/scap-helm part, as the newer calico versions use kubernetes for that info
[13:41:08] <akosiaris>	 but we will probably be able to populate that info from puppet 
[13:41:16] <akosiaris>	 not promising though
[13:42:05] <ottomata>	 hm aye
[13:42:06] <ottomata>	 cool
[13:43:44] <ottomata>	 akosiaris:  eventgate-main in eqiad/codfw will produce to either kafka main cluster in either dc
[13:43:47] <ottomata>	 are the networking rules different?
[13:44:03] <ottomata>	 or should I make rules aallowing both?
[13:44:29] <akosiaris>	 both
[13:44:36] <ottomata>	 k
[13:44:38] <akosiaris>	 so you can switchover in case of need
[13:45:09] <ottomata>	 hm
[13:45:31] <ottomata>	 it doesn't really work that way.  we might coudl do that
[13:45:44] <ottomata>	 but that would make things really complicated with mirror maker replication
[13:45:58] <ottomata>	 eventgate-main in eqiad will always produce to kafka main-eqiad
[13:46:02] <ottomata>	 and vice versa
[13:46:17] <ottomata>	 a switchover will be for eventgate-main
[13:46:19] <akosiaris>	 we might need to
[13:46:23] <akosiaris>	 that's my point
[13:46:30] <ottomata>	 aye ok, ya just networking rules
[13:46:30] <ottomata>	 k
[13:46:31] <ottomata>	 :)
[13:49:34] <fsero>	 ottomata: could you create a phab task if its not much trouble? so we can track this more or less :)
[13:49:46] <fsero>	 or ref an existing one
[13:49:51] <fsero>	 adding serviceops
[13:49:59] <ottomata>	 fsero:  that's the one we've been  talking on: https://phabricator.wikimedia.org/T218346
[13:50:16] <ottomata>	 but i will update description because there isn't one!
[13:50:58] <fsero>	 thanks :)
[13:52:45] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata)
[13:54:04] <ottomata>	 ok patch updated.
[13:54:18] <ottomata>	 proceeding with eventgate -n analytics to codfw + eqiad
[14:06:35] <tarrow>	 akosiaris: hey! I don't suppose you had a chance to look at the wikidata termbox ssr deployment? Just wanted to check if there are more steps form WMDE's side that need doing? It would help us loads with planning if we knew what more you might need from us. We'd thought we were basically there but clearly we overlooked the OpenAPI spec and _info during the week before last
[14:07:50] <akosiaris>	 tarrow: not yet unfortunately. But I 've almost fully caught up with my backlog and will have something for you by tomorrow
[14:08:33] <tarrow>	 akosiaris: coolio! We're trying to get our ducks in a row for tasks we might have missed :) Thanks!
[14:21:00] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Ok, working on two remaining tasks here:  1. Replace chart=eventgate...
[14:59:23] <ottomata>	 akosiaris:  should I get someone on traffic to do the LVS port change? 
[15:15:35] <fsero>	 ottomata: IMHO is not needed i think we can manage it
[15:15:37] <fsero>	 :)
[15:16:00] <fsero>	 i would introduce the main first since is a new service and then move the analytics one
[15:16:49] <ottomata>	 fsero:  ok.  the analytics one is deployed now and ready
[15:17:01] <ottomata>	 I'm fine doing it either way
[15:40:08] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10fsero) @Ottomata sure, i can help you with that
[16:02:45] <wikibugs>	 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Build Thumbor packages for buster - https://phabricator.wikimedia.org/T221562 (10jijiki)
[16:04:42] <wikibugs>	 10serviceops, 10Operations, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10jijiki)
[16:04:45] <wikibugs>	 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Build Thumbor packages for buster - https://phabricator.wikimedia.org/T221562 (10jijiki) 05Open→03Resolved @Gilles All packages have been rebuilt and added to buster-wikimedia main repo. Please reopen if we have any issues.
[16:16:19] <fsero>	 ottomata: 
[16:16:23] <fsero>	 are you around?
[16:25:34] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10fsero) @Ottomata i've created the namespaces and the users for eventgate-main...
[16:35:21] <wikibugs>	 10serviceops, 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, and 4 others: Modern Event Platform: Deploy instance of EventGate service that produces events to kafka main - https://phabricator.wikimedia.org/T218346 (10Ottomata) Awesome thank you!  I'll get that set up today, and we can talk abou...
[16:59:22] <ottomata>	 fsero:  ya
[16:59:25] <ottomata>	 sorry was in meetings
[16:59:32] <ottomata>	 just saw your ping
[16:59:40] <ottomata>	 fsero:  i am trying 
[16:59:41] <ottomata>	 CLUSTER=staging scap-helm eventgate-main install -n main -f main/staging-values.yaml stable/eventgate
[16:59:43] <ottomata>	 and getting
[16:59:47] <ottomata>	 Error: could not find tiller
[17:26:15] <fsero>	 Damn I need to run ottomata 
[17:26:21] <fsero>	 I'll take a look
[17:26:30] <fsero>	 It should work
[17:26:40] <fsero>	 It might be a problem with tiller
[17:29:30] <ottomata>	 ok thanks fsero 
[21:26:09] <wikibugs>	 10serviceops, 10Gerrit, 10Operations, 10Patch-For-Review: Convert Gerrit to use H2 as the database - https://phabricator.wikimedia.org/T211139 (10Paladox)