[00:18:48] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10Jdforrester-WMF) [00:19:15] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10Jdforrester-WMF) [06:29:12] Yup, I'll arrive late into our async meeting will be around at 10 UTC+2 , in any case I did the groom I'm capable of :) so feel free to move things without me [06:44:02] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10hashar) `contint1001.wikimedia.org` is running Jessie and we haven't backported PHP 7.2 to it. php is probably only used for the small website https://integration... [06:48:32] 10serviceops, 10Operations, 10Wikibase-Containers, 10Wikidata, and 2 others: Create a wmf production ready nginx image - https://phabricator.wikimedia.org/T209292 (10hashar) 05Open→03Declined This was for #serviceops! But I decline the task based on @Ladsgroup comment which it would be better to have... [06:57:23] <_joe_> I'm thinking we need one board for the incoming/untriaged tickets [06:58:04] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10Joe) Yes, the burden of maintaing a backport just for contint is not justifiable IMHO. I'm declining the ticket, but if this is blocking something important, plea... [06:58:20] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10Joe) 05Open→03Declined [06:59:34] 10serviceops, 10Operations, 10PHP 7.2 support, 10Patch-For-Review: Socket Errors on PHP7 - https://phabricator.wikimedia.org/T224538 (10Joe) a:03jijiki [07:01:31] instead of backlog? [07:02:07] <_joe_> yeah, so what fsero did (and it seems like a good thing in principle) is to create two "project columns" [07:02:35] <_joe_> I would rather use separate workboards to organize work on large projects, and have a single "goal work" column instead [07:02:38] we could take a quick look what other teams are using [07:02:47] and find what fits for us [07:02:54] <_joe_> I don't think it makes sense to look at what non-SRE teams are doing [07:03:14] <_joe_> our workflow is radically different and we'd try to fit a square in a circle [07:03:14] eg radar is a common column [07:03:31] no just to get more ideas [07:04:19] <_joe_> so my proposal was to have an inbox, where tickets go by default [07:04:53] <_joe_> when we've looked at tickets, we can triage them, and move the ones we want to work on this week into backlog [07:06:32] <_joe_> then doing / blocked externally / radar [07:06:58] <_joe_> basically I think we should have a board per goal to slice individual goals [07:07:07] <_joe_> but let's hear what others think [07:07:39] 10serviceops, 10MediaWiki-General-or-Unknown, 10Operations, 10Core Platform Team (PHP7 (TEC4)), and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Joe) a:05Joe→03None [07:08:05] <_joe_> we can even keep the current structure, but I would drop the "blocking others" column, which is basically every ticket [07:08:16] <_joe_> and add a "Inobx" column as default [07:08:21] I am almost near a laptop [07:08:29] oh and an [07:08:38] externally blocked [07:08:55] ok there is a thing in phab [07:09:25] we cant delete columns, I remember riccardo saying that phab goes insane a bit [07:11:03] <_joe_> uhm I did it already [07:11:06] <_joe_> in the past [07:23:03] <_joe_> so my proposal would be to rename add a "this week" column where we move some of the highest-priority tickets every week [07:23:28] <_joe_> so that we know we have to do those things this week [07:23:35] <_joe_> other than goal work [07:24:44] <_joe_> https://www.mediawiki.org/wiki/Phabricator/Project_management is also relevant, reading it [07:30:14] <_joe_> I changed the sort order so we see untriaged tickets at the top [07:31:36] ok on a laptop [07:32:05] given that we will be able to plan something like 50% of our time [07:33:17] I can see the 'this week' falling behind [07:33:50] <_joe_> if so, tickets still there will be counted when adding new ones [07:35:35] 10serviceops, 10Operations, 10HHVM, 10Performance-Team (Radar), 10User-Marostegui: Increased instability in MediaWiki backends (according to load balancers) - https://phabricator.wikimedia.org/T223952 (10Joe) 05Open→03Resolved I've looked back in the last week or so and we don't see those kind of ins... [07:36:09] _joe_: I have a different proposal based on what you said [07:36:52] maybe we shouldnt have specifi columns for php7 and the pipeline [07:37:03] as we have the blocking and ext blocked as well [07:37:15] I think what you said about a goals column [07:37:23] makes more sense [07:37:33] <_joe_> well I'd wait for others to be around before we decide [07:38:10] It could work that way too [07:38:32] I created the columns per goal to have some visibility about amount of scheduled project work [07:38:56] I think an inbox column is a good idea as it signals untriaged tasks [07:39:11] The column I dislike the most is radar [07:39:52] Either is a task I need to be aware of either I don't want to see it, phab notifications and subscription could help with following tasks that interests me [07:42:39] <_joe_> yeah the jury is out on that, but it's also a way for me to make the all the team be aware of incoming shitstorms :) [07:43:33] <_joe_> so, I also ordered things by priority [07:43:50] <_joe_> should we start by triaging the untriaged tickets in backlog? [07:45:07] Those columns are pretty confusing to me [07:45:33] <_joe_> what is confusing? [07:46:04] <_joe_> we're trying to make things work for everyone, so it's good if we can make it better [07:46:18] Backlog, Next, Doing are time related, but PHP7 and Deployment pipeline are project related, and the exernally blocked/blocking others are "block related" [07:46:27] and yet a task can be in only 1 of these columns [07:46:53] can't something be externally blocked, about php7 and doing next? [07:47:00] <_joe_> uhm [07:47:04] and I am not even sure what radar is about [07:47:08] <_joe_> ok so [07:47:20] <_joe_> as I said, the goal columns should really be workboards [07:47:40] <_joe_> so we should create long-term-project workboards if we like [07:48:00] given that goal work [07:48:06] hmm [07:48:11] I agree with alex [07:48:18] <_joe_> one thing we could do is to consider those "goal backlogs" [07:48:31] <_joe_> things we need to do for those projects and we still haven't done [07:49:08] <_joe_> radar is usually "things we want to be aware of/follow but don't need actual engineering work on our side" [07:49:39] <_joe_> but I'm ok with removing it [07:50:03] it might make sense to have multiple workboards (I think that will require multiple subprojects though?) [07:50:07] <_joe_> it has a value to separate goal-related tasks from other general incoming tasks [07:50:15] tbh, I am not even sure what we are really trying to achieve [07:50:25] <_joe_> a visibility on what needs to be done [07:50:33] <_joe_> and trying to be more orderly about it [07:50:39] but goals are something that we are doing [07:50:51] ok how is this [07:50:52] <_joe_> I keep finding tickets we weren't aware of rotting for months without us knowing [07:50:56] goals column [07:51:03] with the backlog of goals [07:51:40] and and when something is goal related, and in this week's menu [07:51:46] move it to the relevant goal [07:51:54] <_joe_> akosiaris: if you have better proposals than trying to organize the board and deciding on things we do next every week, I'm open to it ofc [07:52:24] For me blocked is an status of the task [07:52:26] Stalled [07:52:26] I feel that having more than one board, might be confusing [07:52:36] I think we need to start with the obvious system limitation. "A task can be in only one of these columns" [07:52:52] so those columns should represent state changes [07:52:57] that sounds good to me [07:53:05] states and the moves state changes* [07:53:23] I think we need jira [07:53:25] * jijiki runs [07:53:28] that again might mean we need more workboards ofc. Which is sad, but anyway [07:53:37] <_joe_> ok so "general backlog" "$goal backlog" -> "next up" -> "doing" -> "done / blocked on others" [07:53:51] <_joe_> the last two are separated ofc [07:54:00] I am not sure we even need the done column [07:54:01] <_joe_> fsero: "stalled" means no one is working on the ticket [07:54:11] <_joe_> the done column is resolving the ticket [07:54:12] next up -> will do soon/this week? [07:54:13] <_joe_> and it disappears [07:54:15] cause we have the "Resolved" status anyway [07:54:31] it feels like duplication [07:54:35] <_joe_> +1 [07:54:41] <_joe_> I was describing the state machine [07:55:09] ok then [07:55:10] <_joe_> so "radar" was, in my idea, for tickets that are not directly actionable for us but we need to keep an ey on [07:55:12] Yup [07:55:34] <_joe_> we want a single goal backlog column [07:55:37] <_joe_> or one per goal? [07:56:13] ok theh resolved status [07:56:25] I think that when we mark a task as resolved [07:56:30] it vanishes, right? [07:56:39] <_joe_> yes [07:56:41] so our case is if a task is resolved from our end [07:56:48] but not resolved [07:56:55] due to other blockers [07:57:31] another question is what to do about the backlog column. We can reasonably expect that almost all tickets will start their life in that column [07:57:50] but it would be a mess waiting to happen if we leave them there lingering [07:58:08] we will lose quite quickly the visibility of new tasks [07:58:12] <_joe_> akosiaris: my proposal was to triage untriaged tickets, and see the old ones [07:58:23] <_joe_> adapt their priority [07:58:31] <_joe_> and then every week pick the most urgent ones [07:58:32] wait wait, triage is a definition concept though [07:58:37] in phabricator that is [07:58:46] <_joe_> setting priority [07:58:48] <_joe_> yep [07:58:49] untriaged tasks are just tasks without a priority [07:58:54] which is completely orthogonal [07:59:17] and also more often changed by external parties due to misconceptions about how it works [07:59:20] <_joe_> what we can also do is have a query to find all tasks that had our tag added in the last week [07:59:47] <_joe_> that's why I proposed one of us does the grooming before we meet on monday [07:59:50] I would avoid having priority somehow represented directly in the columns [07:59:57] <_joe_> I do this with TechCom, and works well [08:00:16] <_joe_> akosiaris: I ordered them by priority rn so we can see what needs to be triaged first [08:00:50] ah yes, then there's that too. We already have it anyway per column, no need to have it represented any other way [08:01:16] <_joe_> yeah [08:01:26] Backlog proposal: every ticket starts there sort the column by date so new issues appear first and every Monday somebody reviews it as _joe_ proposes [08:01:43] Backlog will grow huge but is the way it is [08:01:51] <_joe_> fsero: I think it's easier to have a saved query [08:02:01] So every Monday we need to pick the issues we are going to accept [08:02:16] <_joe_> fsero: it will become huge and we can use that to show we're not enough to do all the incoming work [08:02:55] And if we follow a kanbany approach we should avoid to put new issues until the actual ones are closed [08:03:23] <_joe_> yeah, no, we're not going full kanban-y [08:03:32] let's do scrum [08:03:33] :P [08:03:39] ah no, flow [08:03:42] jijiki: ;-) [08:03:43] <_joe_> that approach assumes there is on average the ability to process the incoming requests [08:03:50] flow flow [08:03:55] I have to say I am a certified flow specialist already [08:03:55] please let's do flow [08:03:56] :P [08:04:10] <_joe_> if flow a real thing or just a mockery? [08:04:20] well... it depends? [08:04:30] akosiaris: joel said that the video was too depressing to watch [08:04:30] I fear someone is going to take it more seriously than they should [08:04:45] what? It was the best keynote I ever saw!!! [08:04:52] I was laughing all the way [08:04:53] _joe_: it depends on your sarcasm-o-meter [08:05:16] anyway, back to the columns, let's not get derailed [08:05:20] <_joe_> yeah [08:05:40] <_joe_> so should we have a column for all the k8s/pipeline work backlog? [08:05:47] lemme ask a different question. Is a single workboard enough for what we want to do [08:05:48] ? [08:06:05] <_joe_> for the basics, yes. I guess each goal work might need more workboards [08:06:17] <_joe_> but I fear having more workboards than people won't help [08:06:22] If we want to represent the subteam work it should start with one [08:06:36] It would be messy but is better than several [08:06:44] <_joe_> so for instance I like that service requests have their workboard [08:06:59] <_joe_> and that's why I'd make a single backlog column of all the k8s work [08:07:25] <_joe_> not just for service requests [08:07:35] I would like a single column [08:07:47] <_joe_> for what? [08:07:51] we are very few, lets not complicate it as if we were an 8 people team [08:07:55] lol sorry [08:07:57] board [08:07:59] single board [08:08:11] <_joe_> yeah, it's not going to be perfect [08:08:17] <_joe_> and we can change it easily [08:08:24] I can live with not perfect [08:08:30] <_joe_> it's not like production is running on top of it [08:09:29] lol [08:09:56] lets try KISS, and iterate if we have issues [08:10:14] just go for a "incoming/backlog" => "Next up" => "Doing" I 'd say [08:10:19] I would rename Next to "Try to do this week but no promises" [08:10:44] <_joe_> be bold [08:10:45] akosiaris: +1 [08:10:46] don't put time constraints in the column names, it won't help [08:11:01] my only question then is [08:11:03] akosiaris: yeah, we know [08:11:21] do we want to have "incoming/backlog" as empty as possible or not? [08:11:32] <_joe_> it's never going to be, no [08:11:33] we now that it wont happen [08:11:46] <_joe_> it's going to grow, we're severely understaffed [08:11:47] yeah it is probably pointless [08:11:50] but putting a lower priority [08:12:06] use the 'low' priority, and not have anyone being offended by it [08:12:09] would help [08:12:22] <_joe_> jijiki: who cares, as long as it's fair? [08:12:27] ok then. How about triaging then? [08:12:43] _joe_: I remember what happend last time I marked a task as invalid :) [08:12:43] will we be triaging (setting priorities) on the backlog column? [08:12:46] <_joe_> akosiaris: first thing I'd ask you is to move k8s related things to the column [08:12:54] what column? [08:13:05] <_joe_> akosiaris: do you have the workboard open? [08:13:21] <_joe_> we have 3 backlog columns, 1 general, 1 k8s, 1 php7 [08:13:21] yes and I said I don't like it and finding it confusing [08:13:29] me too [08:13:47] <_joe_> so you want a single huge backlog? [08:13:58] <_joe_> I find that confusing, but ok :) [08:14:06] I don't see how a goal specific backlog would help in the context of this workboard [08:14:13] we could go to the goals column [08:14:38] <_joe_> I think it allows to see the balance between goal work and various incoming random tasks [08:14:42] especially since we are close to be a "one man team per goal" [08:14:51] akosiaris: I would put it like [08:14:56] we will just end up each managing their own column [08:14:58] which is ... meh [08:14:58] a goal task with normal priority [08:15:14] is more important than a general backlog task with normal priority [08:15:23] that is how I see it [08:15:26] <_joe_> that was my point, yes [08:15:52] <_joe_> anyways [08:15:56] akosiaris: should we try it and see how it goes? [08:16:08] define it again please? [08:16:08] <_joe_> let's move the goal tasks back to the backlog, it's 17 of them currently [08:16:22] <_joe_> we can re-add them if we miss them [08:16:33] akosiaris: normal priority tasks in the goals column [08:16:49] are more important than normal priority tasks in the backlog column [08:16:56] a single goals column ? [08:16:57] <_joe_> btw, I'd prefer "project" to "goal". But that can be represented in a separate workboard to get better detail [08:17:24] _joe_: programs :P [08:17:30] [08:17:33] <_joe_> let's start as simple as possible, and we refine it, given there is no consensus [08:17:55] <_joe_> so can someone move the goal tickets back to the backlog and hide those two columns? [08:17:55] so... "incoming/backlog" => "Next up" => "Doing" ? [08:18:09] are we agreed on the above ^ ? [08:18:11] <_joe_> yes, and 'externally blocked' and 'radar' [08:18:26] <_joe_> we need to have a place where things we're blocked on by other teams go [08:18:45] ok ok about that, but 'radar' ? [08:19:05] <_joe_> so let's make an example [08:19:10] let's avoid radar please for now [08:19:13] keeping it simple [08:19:26] "incoming/backlog" => "Next up" => "Externally blocked" => "Doing" [08:19:34] <_joe_> https://phabricator.wikimedia.org/T218217 [08:19:36] ok [08:19:43] <_joe_> this is not direclty actionable by us [08:19:51] <_joe_> but people might need feedback from us [08:20:04] <_joe_> having it in backlog will only make a mess [08:20:04] we should be pinged then? [08:20:14] <_joe_> it's tagged "serviceops" [08:20:23] <_joe_> where should it be if not in "radar"? [08:20:28] it being in "radar" column will not make me look into it more often tbh [08:20:29] <_joe_> that's my question [08:20:37] I like akosiaris proposal but I'll amend them what if we have project tags [08:20:40] notifications will [08:20:45] Serviceops-k8s [08:20:51] Serviceops-php7 [08:20:52] Etc [08:20:57] <_joe_> fsero: yeah that is my idea too [08:20:59] Just for easier querying later [08:21:05] fsero: yup, +1 [08:21:17] but hangon [08:21:18] <_joe_> akosiaris: ok, so, say we don't have the radar column [08:21:23] we have a php7 tag [08:21:25] <_joe_> where does that task fit? [08:21:37] <_joe_> jijiki: which is ambiguous [08:21:48] <_joe_> it includes tasks from 10 different teams I think [08:21:57] if something is servicesops+ php7 [08:21:57] <_joe_> but yes we can query for that plus serviceops [08:22:03] why yet another tag ? [08:22:18] _joe_: well depends. If we are pedantic, almost nowhere and it should be untagged serviceops. [08:22:32] I mean, servicesops+php7=Serviceops-php7 [08:22:35] <_joe_> akosiaris: uhm so your proposal is untagging [08:22:42] if my math is right :p [08:22:49] if we want to keep an eye on it, maybe that workboard is not the best way to do that [08:22:52] <_joe_> jijiki: yeah we agreed on the concept, not the specifics of each tag [08:23:12] <_joe_> akosiaris: radar is a way to keep tickets not immediately actionable out of the backlog, too [08:23:28] <_joe_> untagging tickets is... uhm [08:23:36] _joe_: IMHO it should be outside the board [08:23:41] You cannot work on that [08:23:41] we will lose them forever [08:23:54] As a compromise maybe add a radar tag [08:23:57] and sometimes we might need to jump into a conversation [08:23:59] I can already see serviceops-radar [08:24:00] And have another board for it [08:24:03] <_joe_> ok so [08:24:05] fsero: lol, +1 [08:24:07] <_joe_> that's the alternative [08:24:09] you beat me to it [08:24:11] <_joe_> we add a new tag [08:24:17] <_joe_> it's ok [08:24:20] <_joe_> so [08:24:42] <_joe_> for now leave the column, and move there tickets, I will create a Serviceops-Radar tag and we can retag them [08:25:27] <_joe_> while you all work on moving tickets around columns according to what was said, I'll see how to create a tag like this (I'm a phab project creator but I need to follow policies) [08:26:25] So summary [08:27:03] We would have the following columns: backlog, next up,doing, blocked by others [08:27:07] Done? [08:27:11] +1 [08:27:41] <_joe_> https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects/Project_types [08:27:50] I hid the k8s/php7 columns and moved tasks back to backlog [08:27:59] Radar column will disappear and we would retag those tasks with serviceops-radar [08:28:08] I left radar as is while _joe_ creates the serviceops-radar tag [08:28:11] <_joe_> I think radar is best served as a subproject, yes [08:28:21] <_joe_> yeah lemme deal with bureaucracy [08:28:45] not knowledgeable about the intricacies of the subproject idea, but sure, let's give it a try [08:29:25] And we will decide/notify on Monday meetings which task is going to be offered to the kanban gods that week [08:29:34] ? [08:29:38] <_joe_> so one thing we need to do now [08:29:54] <_joe_> is someone takes up formalizing things we said here in some wiki page [08:30:10] <_joe_> very broadly and coarsely for now [08:30:20] * _joe_ puts on agile hat [08:30:22] or in https://phabricator.wikimedia.org/project/profile/3775/ [08:30:40] keep the docs close to the implementation ;-) [08:30:41] <_joe_> we should also come up with definitions for what various priorities mean for us [08:30:47] * _joe_ puts the hat away [08:31:13] * jijiki puts the harry potter sorting hat [08:32:00] ok I will try to triage [08:32:04] the backlog [08:32:17] * fsero fsero mumbles slytherin [08:32:33] 10serviceops, 10Release-Engineering-Team-TODO, 10Scap, 10User-jijiki: Deploy scap 3.10.0-1 - https://phabricator.wikimedia.org/T224915 (10jijiki) p:05Triage→03Normal [08:32:36] it's always hufflepuf, you know that :P [08:32:43] haahha [08:33:08] 10serviceops, 10Operations, 10wikitech.wikimedia.org, 10PHP 7.2 support, 10Patch-For-Review: switch wikitech to PHP 7.2 - https://phabricator.wikimedia.org/T223393 (10jijiki) p:05Triage→03Low [08:33:13] if you disagree with a priority, ping [08:33:46] 10serviceops, 10Citoid, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Figure out how to test Citoid with Zotero in the pipeline - https://phabricator.wikimedia.org/T225236 (10jijiki) p:05Triage→03Normal [08:34:28] 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO: Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10jijiki) p:05Triage→03Low [08:37:17] anyone has an input for https://phabricator.wikimedia.org/T221315 [08:37:23] Determine future of bare-metal hosting for services like WDQS, ElasticSearch, RESTBase Cassandra, etc. [08:37:51] https://phabricator.wikimedia.org/project/manage/3775/ [08:37:55] docs updated [08:38:09] feel free to amend [08:38:23] * jijiki amends blubberoid [08:38:34] I put only the serviceops-radar subproject/tag in the list but I expect it to grow [08:38:50] 10serviceops, 10Discovery-Search, 10Elasticsearch, 10RESTBase, and 5 others: Determine future of bare-metal hosting for services like WDQS, ElasticSearch, RESTBase Cassandra, etc. - https://phabricator.wikimedia.org/T221315 (10jijiki) p:05Triage→03Low [08:38:52] also [08:38:56] we could agree [08:38:57] <_joe_> "Consequently, when you add the first subproject to an existing project, all of the project's current members are moved to become members of the subproject instead. Implicitly, they will remain members of the parent project because the parent project is an ancestor of the new subproject." [08:39:00] <_joe_> sigh [08:39:09] <_joe_> I will try and do it, let's see what happens [08:39:20] that every week we will pick a couple of low priority tasks if time permits [08:39:22] <_joe_> please stop editing the workboard for a few minutes [08:39:27] ok [08:39:30] I am stopping [08:41:27] _joe_: tell me when [08:41:31] 10serviceops: Investigate outgoing discarded packets in the codfw kubernetes cluster - https://phabricator.wikimedia.org/T226237 (10akosiaris) [08:43:03] <_joe_> https://phabricator.wikimedia.org/project/profile/4114/ [08:43:21] <_joe_> I'll start retagging the radar column [08:43:34] 10serviceops: Investigate outgoing discarded packets in the codfw kubernetes cluster - https://phabricator.wikimedia.org/T226237 (10akosiaris) p:05Triage→03Low https://grafana.wikimedia.org/d/PRA2F67Zz/t226237?orgId=1 was created to help debug with this. It makes more clear that this are indeed outgoing ICMP... [08:46:21] <_joe_> well most things in radar were actually backlog items :/ [08:50:17] 10serviceops, 10Operations, 10Operations-Software-Development, 10Patch-For-Review, and 3 others: Convert makevm to spicerack cookbook - https://phabricator.wikimedia.org/T203963 (10akosiaris) Should we close this? Is there anything left to be done? [08:50:49] <_joe_> akosiaris: retag that to radar [08:53:03] should wikibugs handle serviceops-radar stuff in this channel? I think not fwiw, but feel free to disagree [08:53:14] <_joe_> I think not as well [08:55:09] looking into the backlog column now, half of it can be tagged serviceops-radar :P [08:55:19] <_joe_> jijiki: I'll create a PHP7 milestone [08:55:36] <_joe_> akosiaris: which is interesting as well [08:55:58] <_joe_> I added a column "watched" for things we're currently actively looking at [08:56:18] <_joe_> so there is also a backlog of things we should interact on and we're not [08:56:35] watched column? where? [08:56:44] <_joe_> in the radar workboard [08:56:45] * akosiaris hides radar column btw [08:56:53] _joe_: ah ok, makes sense [08:57:27] <_joe_> I think at this point we need to generate data other than "I work 60 hours a week" to show we have a problem [08:57:55] <_joe_> oh and ofc, there are tons of tasks we have to work on that are not tagged properly [08:58:01] <_joe_> we should start doing that [08:58:27] <_joe_> (tagging tasks I mean) [08:59:08] <_joe_> can we talk shortly about priorities? We have 5 priority levels [08:59:18] <_joe_> ubn!, high, normal, low, lowest [08:59:49] <_joe_> can we come up with shared definitions for all of them, and try to apply those categories to our tasks? [09:00:22] <_joe_> like: ubn! is easy - stop whatever else you're doing that is not ubn, and to it [09:00:46] <_joe_> so ubn == incident [09:01:19] <_joe_> the big issue for me is, below that, granularity is hard. [09:02:31] <_joe_> I think it's easier to think in terms of "highest prio come first" [09:02:57] <_joe_> something that is low priority today might be high priority in two months, its priority should be adapted correspondingly [09:04:17] https://phabricator.wikimedia.org/H323 fwiw [09:04:36] <_joe_> oh thanks I intended to do it later [09:05:25] <_joe_> jijiki: ok, how do you plan to see the php7-related tasks on a workboard? I don't think you can create a workboard for combined tags [09:05:51] <_joe_> akosiaris: are you moving tasks to radar from the backlog? [09:06:21] well yeah, but only the easy pickings for now [09:06:51] _joe_: I have my own board as well, so even they are not finely grained on our workboard [09:06:57] I think it will work [09:08:06] 10serviceops, 10Release Pipeline, 10docker-pkg, 10Patch-For-Review: Remove backports from wikimedia-jessie - https://phabricator.wikimedia.org/T219580 (10Joe) [09:08:08] 10serviceops, 10Release-Engineering-Team, 10Patch-For-Review: Rebuild integration/config images based on jessie - https://phabricator.wikimedia.org/T219748 (10Joe) 05Open→03Stalled p:05Triage→03Normal The situation is, release engineering doesn't feel confident doing general upgrades like this, and a... [09:09:21] 10serviceops, 10Release-Engineering-Team, 10Scap: Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10Joe) [09:10:42] <_joe_> jijiki: you should de-assign tickets you're not working on [09:11:10] if there are tasks I will do [09:11:23] should I keep them assigned? [09:11:28] I believe I should [09:11:50] and sometimes people ping me by assigning tasks, which is good [09:15:37] <_joe_> no it's a very bad practice [09:15:48] <_joe_> esp if those are things you're not actively doing [09:15:54] <_joe_> we do that all the time [09:17:53] it works for me so far [09:18:47] let's see how it goes and adjust it again later? [09:18:55] <_joe_> my example is https://phabricator.wikimedia.org/T218328 [09:19:07] oh that is a special case [09:19:09] sure [09:19:20] 10serviceops, 10Scap: Scap2 to use etcd for target servers - https://phabricator.wikimedia.org/T218328 (10jijiki) a:05jijiki→03None [09:19:27] but I am talking about eg thumbor tasks [09:19:30] <_joe_> frankly I don't know what to do with those tags [09:19:38] <_joe_> sure, you're still handling those [09:19:44] <_joe_> not working on them "right now" [09:19:51] <_joe_> but they're still your responsibility [09:25:35] You should not have a task tailored to an individual. Probably you will end up doing it but is a good practice to describe what is needed to do in that task so another one can take it and do it [09:29:55] _joe_: lets leave it to each one individualy for now [09:57:26] _joe_: akosiaris so we leave it as is [09:57:31] I can triage more backlog [09:57:39] and checkin again on monday ? [09:58:12] sure [09:58:48] <_joe_> I would like to make some triage, there are a couple quick wins I could pick for next week [09:59:17] <_joe_> I'll triage the tickets I know more about [09:59:23] <_joe_> I invite you all to do the same [09:59:51] 10serviceops, 10Release-Engineering-Team-TODO: Our docker base images lack tags - https://phabricator.wikimedia.org/T218342 (10Joe) p:05Triage→03Normal [10:01:26] 10serviceops, 10Packaging: Guidelines for Rust/Go tools deployment - https://phabricator.wikimedia.org/T220836 (10Joe) p:05Triage→03Low [10:02:55] _joe_: I wil triage after lunch if you don't mind [10:02:58] and sync up later [10:03:01] works? [10:03:10] <_joe_> that's ok, if everyone of us does some of it [10:03:20] * jijiki lunch [10:05:20] 10serviceops, 10Packaging: Guidelines for Rust/Go tools deployment - https://phabricator.wikimedia.org/T220836 (10Joe) While we wanted to get into a discussion about go packaging at the offsite, we had no time for it. My current proposal for it would be the following: - Use "go mod" under golang >= 1.11 to c... [10:08:52] 10serviceops, 10Cloud-VPS, 10Patch-For-Review, 10Puppet, and 2 others: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10Joe) p:05Triage→03Low [10:11:11] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10Joe) p:05Triage→03Low [10:13:04] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10Joe) p:05Low→03Triage [10:13:21] <_joe_> meh moving to different places on the board changes priority [10:13:41] 10serviceops: Find which machines will be over 5 years old during FY19-20 - https://phabricator.wikimedia.org/T217764 (10Joe) p:05Triage→03Low [10:15:03] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: etcd1004-1006 is unused and idle, use the cluster or kill it. - https://phabricator.wikimedia.org/T212934 (10Joe) p:05Triage→03High [10:17:54] 10serviceops, 10Operations, 10observability, 10PHP 7.2 support, and 2 others: [Regression] fatal-errors.php action=segfault results in a 503 error under php7-fpm. - https://phabricator.wikimedia.org/T223336 (10Joe) [10:19:13] 10serviceops, 10MediaWiki-Logging, 10Operations, 10Wikimedia-Logstash, and 8 others: Port mediawiki/php/wmerrors to PHP7 and deploy - https://phabricator.wikimedia.org/T187147 (10Joe) [10:25:13] 10serviceops, 10MediaWiki-Logging, 10Operations, 10Wikimedia-Logstash, and 8 others: Port mediawiki/php/wmerrors to PHP7 and deploy - https://phabricator.wikimedia.org/T187147 (10Joe) @Legoktm kindly did my job and created an [[https://salsa.debian.org/mediawiki-team/php-wmerrors | upstream package ]] for... [10:29:01] <_joe_> anyone else sees an issue in our board? [10:29:35] <_joe_> so for one I would not put goal tickets in the "doing" column [10:29:52] <_joe_> as in the actual "main ticket task" [10:30:00] <_joe_> but maybe I'm wrong [10:33:28] 10serviceops: Investigate outgoing discarded packets in the codfw kubernetes cluster - https://phabricator.wikimedia.org/T226237 (10akosiaris) After some mangling with iptables trying to figure out what is going on I 've managed to capture these packets (and their drops?) in iptables and log them ` Jun 21 10:27... [10:36:26] _joe_: the umbrella tasks? [10:37:00] yeah we can skip them I guess. Not that I mind them anyway in doing, although they do clutter either backlog column or doing column [10:47:41] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Helm packages deployment tool, at least for cluster applications. - https://phabricator.wikimedia.org/T212130 (10fsero) a:03fsero [12:06:16] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, 10User-jijiki: Investigate systemd hardening to replace Firejail for Thumbor - https://phabricator.wikimedia.org/T212941 (10jijiki) p:05Normal→03Low [12:06:18] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, 10User-jijiki: Investigate systemd hardening to replace Firejail for Thumbor - https://phabricator.wikimedia.org/T212941 (10jijiki) p:05Low→03Normal [12:06:21] 10serviceops, 10Operations, 10Thumbor, 10Wikimedia-Logstash, 10User-jijiki: Stream Thumbor logs to logstash - https://phabricator.wikimedia.org/T212946 (10jijiki) p:05Normal→03Low [15:10:07] _joe_: akosiaris i think i've addressed your nits https://gerrit.wikimedia.org/r/c/operations/puppet/+/517888 [15:11:30] <_joe_> fsero: I'll take a look in a few [18:28:28] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10Jdforrester-WMF) OK, will switch out with T224591. [18:28:41] 10serviceops, 10Continuous-Integration-Infrastructure: Provide php72 on contint1001 rather than php56 - https://phabricator.wikimedia.org/T226224 (10Jdforrester-WMF) [20:50:11] 10serviceops, 10Release-Engineering-Team, 10Scap, 10PHP 7.2 support, and 2 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5268558, @Joe wrote: > To summarize the situation a bit, we have 2 problems right now that need to...