[00:36:50] https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/544066#message-779bbe18a259722b3cd55fb04da11fed0cabad84 [13:25:21] <_joe_> hey everyone in serviceops - you should add your OKRs - at least the one you're working on - to the status document [13:27:42] <_joe_> lemme ping you all apergos akosiaris effie mutante rlazarus ^^ [13:28:58] not supposed to add them, according to mark, because contractor [13:29:24] we will have a separate checkin outside of this meeting [13:30:27] we should? [13:30:50] <_joe_> akosiaris: that's what joel was suggesting AFAICT [13:31:37] I am not sure at all tbh [13:31:48] that OKRs thing in the email is since 3 weeks ago [13:31:53] when OKRs weren't even decided [13:32:11] not to mention the fact that the document will become instantly unyieldy [13:32:18] as if it's not big enough already [13:33:13] it's anything from 52 to 78 OKRs and that's assuming we only have 2 or 3 each [13:33:36] <_joe_> Foundations has done something along those lines [13:33:49] <_joe_> I am working on one only for now, officially at least [13:34:31] really? AFAOCT the status from the Foundations sub meeting on Wednesday was that this isn't really set in stone how we do the reporting (and where, as there's also Betterworks) [13:34:48] I have the same feeling [13:34:52] <_joe_> moritzm: I am looking at the document [13:35:08] I can see the document does have Os and KRs in it [13:35:48] <_joe_> akosiaris: how would you be reporting on your work otherwise? [13:35:56] I get btw that they should be somewhere better than betterworks [13:36:27] in the sub team meetings [13:36:43] doing it the bigger SRE meeting isn't going to scale [13:37:15] <_joe_> moritzm: don't tell me, tell the rest of your team [13:37:23] sigh OKR situation is still nebulous [13:37:38] I have the obvious question as well... we used to track stuff in phab tasks [13:37:40] so far I only see updates from Riccardo and Arzhel, that's hardly the whole team :-) [13:37:44] what about now? [13:38:08] I think the most reasonable way to address this is to hold updates for now and discuss how we proceed on Monday [13:38:24] I could imagine a setup along the lines of: [13:38:37] - discuss specific progress in sub team meetings [13:39:08] - and only mention things in the Monday meeting which are actually cross-team relevant [13:39:14] but who knows :-) [13:40:37] yeah I would expect summarizations to make it to the Monday meeting [13:40:52] I mean we could write everything down and only bold the interesting parts [13:41:01] or we copy the progress reports from the sub team meetings to a common weekly doc, but doing it twice seems fiddly and error-prone [13:41:05] and thankfully it's not real paper (otherwise I would be screaming) [13:41:18] but what's the point if nobody is going to read it? [13:42:16] fwiw, the summary approach is what I've done in the past and it works reasonably well -- the subteam probably wants a different kind of detail than the broader team anyway [13:42:40] reasons to spend time in the big meeting include, like, "we're going to be late with KR X because we were held up by Y, this might impact you because Z" [13:43:00] yeah that makes sense to me [13:43:26] paravoid: got any opinions ^ ? or should we be discussing it on Monday? [13:43:46] <_joe_> rlazarus: yes that makes sense, but truth is we've yet to figure out how to structure the meeting now [13:44:00] or our phab tasks [13:44:02] <_joe_> rlazarus: the idea of bolding the items you want to discuss [13:44:11] <_joe_> is that of basically doing a summary only [13:49:50] sorry, what's the question? [13:52:42] OKRs. Do they make it as is in the Monday meeting document? [13:52:53] do we summarize them in some form and put them three [13:52:57] something else entirely? [14:28:42] 10serviceops, 10Operations, 10Performance-Team, 10Wikimedia-General-or-Unknown: Investigate recurrent latency spikes for the MediaWiki appservers - https://phabricator.wikimedia.org/T235872 (10elukey) Custom dashboard to get some metrics about the MW WANObjectCache (derived from the Performance Team's dash... [14:46:52] 10serviceops, 10DBA, 10Operations: Backups on buster hosts fail to run - https://phabricator.wikimedia.org/T235838 (10akosiaris) I see 2 paths forward: * Push forward with the migration in order to fix this * Backport bacula 9 from the stretch backport to jessie and do the upgrade in place on the old hosts... [15:09:58] so do we know what to do ? [15:10:11] my other option would be just the ork I am working on [15:10:28] or do whatever rlazarus proposes [15:11:43] I will trust someone who has done orks before :) [15:40:04] working on a tool for testing apache configs, that sends a list of HTTP requests to a server and makes assertions about the responses -- say, "/wiki/Main_Page" with header "Host: en.wikipedia.org" should return 200, with response body containing "Wikipedia" [15:40:19] any objections to naming the tool "httpbb", short for "http black-box testing"? [15:40:24] counterproposals welcomed :) [15:43:08] <_joe_> I like the name, which is I suspect a bad sign [15:43:16] <_joe_> given the names I come up with for software [15:48:10] <_joe_> it might be a bit late on friday for the greeks to be around [15:48:44] <_joe_> mutante: ping me when you're around, I want to wrap up setting up the LVS/discovery for parsoid-php today [16:01:31] <_joe_> rlazarus: let's use httpbb [16:01:46] sold [16:01:47] <_joe_> operations/software/httpbb it is [16:02:29] <_joe_> https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/httpbb [16:03:30] thanks@ [16:03:32] *! [16:37:46] _joe_: hi! should we merge the change to discovery.yaml? [16:38:55] <_joe_> mutante: you should first add the conftool-data too in that change [16:39:09] i see the comment on gerrit, ack [16:39:11] <_joe_> then enable the corresponding confctl objects [16:39:31] <_joe_> but for now my idea would be you should install mediawiki on all wtp hosts at this point [16:39:41] <_joe_> and set them as pooled=yes in conftool [16:40:07] <_joe_> so that on monday we can hand the keys to the realm to subbu [16:40:22] <_joe_> discovery I can finish for you on monday morning in 10 minutes if needed [16:40:23] i was going to ask that.. install mediawiki on more/all [16:40:29] <_joe_> go with all [16:40:32] <_joe_> :) [16:40:39] <_joe_> I am going afk now, btw [16:40:45] <_joe_> I'm around since ~ 6 am [16:40:53] ok, that sounds like a good deal, i will install more mediawiki [16:41:01] and leave the discovery (but amend to it ) [16:41:23] _joe_, mutante mobrovac i'll schedule a sync up meeting monday ... so we can verify all things are good to go for everyone. [16:41:27] have a good weekend. [16:41:36] so that we have a change for adding it to conftool-data [16:42:49] <_joe_> mutante: +1 [16:43:05] _joe_ / subbu: sounds all good, except on Monday i am also busy in 19:00–22:00 UTC # (Gerrit migration) [16:43:05] <_joe_> subbu: if nothing bad happens, the internal part of the equation will be solved [16:43:28] that is when you usually have parsoid deploy kind of [16:43:29] <_joe_> mutante: I can be at the meeting if you're busy, or we can have it another day [16:44:07] mutante, ok. _joe_ ya [16:44:09] thanks! [16:44:22] just pointing out the regular parsoid deploy slot could be affected by gerrit work [16:44:25] (too) [16:44:44] have a good weekend all [16:44:58] i think mobrovac is blocked on traffic url encoding issues for restbase .. but, mobrovac can enable restbase changes whenever all is green on his end. [16:45:23] we can probably do both, maybe not the exact slots on Deployment calender [17:13:37] amended to the discovery change, adding it in conftool-data/discovery/mediawiki.yaml but that is just 1 line. the data for each node (parsoid, parsoid-php) is already there from previous change [17:27:16] mutante, i created https://phabricator.wikimedia.org/T235902 ... feel free to add any other subtasks related to this. [17:31:28] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) [17:31:38] subbu: ack, done. it's going to be a check box on that [17:32:19] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10ssastry) [17:33:03] 10serviceops, 10Operations: Set up LVS for parsoid/PHP - https://phabricator.wikimedia.org/T233722 (10ssastry) [17:33:39] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) [19:19:16] 10serviceops, 10Operations, 10Release Pipeline, 10Goal, 10Release-Engineering-Team (Pipeline): Self-service Deployment Pipeline - https://phabricator.wikimedia.org/T228676 (10Ladsgroup) p:05Normal→03High [22:10:21] 10serviceops, 10Growth-Team, 10Notifications, 10Operations, and 2 others: Dashboards for monitoring of echostore - https://phabricator.wikimedia.org/T235558 (10Eevans) This is now done: See https://logstash.wikimedia.org/app/kibana#/dashboard/AW3go_uCx3rdj6D8q-he & https://grafana.wikimedia.org/d/IfJykaTZk... [23:20:00] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) With [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/544232 | this change ]] the parameters `profile::parsoid::use_php` and `has_lvs` have become defau... [23:21:14] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) [23:22:43] all wtp servers are now parsoid-php appservers https://phabricator.wikimedia.org/T233654#5588312 | [23:22:46] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=wtp [23:30:52] 10serviceops, 10Operations: Set up LVS for parsoid/PHP - https://phabricator.wikimedia.org/T233722 (10Dzahn) Linking my pending Gerrit changes here: [ ] [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/542572 | discovery.yaml: add parsoid-php microservice ]] [ ] [[ https://gerrit.wikimedia.org/r/c/ope... [23:33:23] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) [23:34:16] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) p:05Normal→03High [23:34:27] 10serviceops, 10Operations, 10Patch-For-Review: Set up LVS for parsoid/PHP - https://phabricator.wikimedia.org/T233722 (10Dzahn) p:05Normal→03High [23:36:42] 10serviceops, 10Operations, 10Patch-For-Review: Make the parsoid cluster support parsoid/PHP - https://phabricator.wikimedia.org/T233654 (10Dzahn) @Joe If from here on the remaining steps are T233722 then this ticket should be resolved i think.