[00:08:08] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) [00:08:23] 10serviceops, 10SRE, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Legoktm) 05Open→03Resolved a:03Legoktm Conclusi... [00:10:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2280.codfw.wmnet'] ` an... [00:10:54] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Legoktm) [00:10:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10Legoktm) 05Stalled→03Open [00:11:30] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2279.codfw.wmnet'] ` an... [00:28:07] 10serviceops, 10SRE, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Investigate possible performance degradation on mediawiki servers after Debian Buster upgrade - https://phabricator.wikimedia.org/T273312 (10Dzahn) Thank you very much for the flamegraphs and fu... [00:43:33] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1318.eqiad.wmnet'] ` an... [00:51:11] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1310.eqiad.wmnet'] ` an... [01:08:27] 10serviceops: Migrate WMF Production from PHP 7.2 to a newer version - https://phabricator.wikimedia.org/T271736 (10Reedy) [06:39:22] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review, 10User-fsero: Set up PodSecurityPolicies in clusters - https://phabricator.wikimedia.org/T228967 (10JMeybohm) >>! In T228967#6801335, @dduvall wrote: > Change to blubber(oid) (https://gerrit.wikimedia.org/r/c/blubber/+/660771) has been deployed. Thanks... [07:25:49] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10elukey) If we keep the backups for conf1* it should be fine, conf2* replicates via etcdmirror from conf100*, so if this unblocks you it should be doable. Let's check with @Joe to be sure :) [07:55:22] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` snapshot1009.eqiad.wmnet ` The log c... [08:31:17] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['snapshot1009.eqiad.wmnet'] ` and were **ALL** successful. [08:52:53] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) snapshot1009 was idle so I converted it. snapshot1010 should become idle in an hour or two, so I'll be able to do that later tod... [09:40:52] jayme: o/ good morning, as FYI I am proceeding with https://gerrit.wikimedia.org/r/c/operations/puppet/+/661687 with Valentin if you are ok [09:41:12] elukey: ack, go ahead [09:41:16] super [09:41:40] ping me if you need help [10:05:40] 10serviceops, 10SRE: upgrade conf2* servers to stretch - https://phabricator.wikimedia.org/T271573 (10jcrespo) Let's make sure this is true, and only canonical data will be on conf100* before moving ahead with that, otherwise I prefer to get blocked and speed up the upgrade. [10:19:27] so curl -v -k https://eventstreams-internal.svc.eqiad.wmnet:4992/_info [10:19:37] and codfw seems to work :) [10:20:27] jayme: ok to move to monitoring_setup ? [10:21:48] Nice! Yeah, switch to monitoring setup then [10:45:34] jayme: monitoring up and green, moving to production if you are ok [10:45:37] (plus dns-disc) [10:46:16] elukey: yeah! [10:46:22] ack! [10:59:18] 10serviceops, 10Prod-Kubernetes: Kubernetes prod and staging share the same cluster key in hiera - https://phabricator.wikimedia.org/T273866 (10JMeybohm) p:05Triage→03Medium [11:09:26] jayme: last one! https://gerrit.wikimedia.org/r/c/operations/puppet/+/661702/ [11:12:12] hahahah [11:12:23] E501 line too long (103 > 100 [11:13:00] you will need a # noqa elukey [11:13:15] yep yep I know :D [11:13:31] I didn't see that coming [11:13:57] teaching from that file, "# noqa" could be "# analytics" :-P [11:14:43] jayme: are you by any chance sending a subtle message about Analytics? :D [11:15:07] nono :D [11:15:19] ahhhh okok [11:15:20] :D :D [11:15:39] I am lucky that Joe is not seeing this conversation [11:15:45] it's not that subtle :-P [11:16:35] yeah, how lucky you are that he is of this morning and this whole conversation will be lost in the limbo of the internet by then :) [11:17:28] jayme: task done finally! Thanks a million for all the help and patience, really appreciated [11:18:57] congrats! :) Sure thing. Nice that you added all those clarifications to the docs in the process! [11:19:31] you're now officially onboarded to serviceops. Please feel free to leave your old team behind :D [11:19:36] * apergos is now taking bribes to make sure this conversation does in fact stay lost :-P [12:48:56] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by ariel on cumin1001.eqiad.wmnet for hosts: ` snapshot1010.eqiad.wmnet ` The log c... [13:25:32] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['snapshot1010.eqiad.wmnet'] ` and were **ALL** successful. [13:25:56] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10elukey) Tested with `ssh -L 4992:eventstreams-internal.discovery.wmnet:4992... [13:32:33] I wonder if we could ever live in a world where "wikidata" was deemed to be a different service than the other mediawiki sites [13:32:46] [13:54:55] <_joe_> addshore: we certainly could, but why would we? [13:55:19] <_joe_> I mean are there compelling reasons that don't include "run on a different version of MediaWiki"? [13:55:51] <_joe_> addshore: I even considered creating a "one service per wiki" paradigm when we'll move to kubernetes [13:56:09] <_joe_> but then deemed it too inefficient given the disparities in traffic we have [14:08:10] _joe_: it could make sense to split out a few of the larger wikis, and then have a 'default' instance -- not unlike what we do for database shards -- but also I agree I don't see any immediate advantages in this case [14:11:51] <_joe_> cdanis: unless we can move to avoid multiversion [14:11:57] <_joe_> that would be a huge advantage [14:12:17] one set of k8s jobs per deployment group? [14:28:24] <_joe_> yes [14:28:33] <_joe_> well, possibly multiple [14:28:53] <_joe_> but also, we'd have to revisit the deployment groups [14:29:20] <_joe_> my current line of thinking is "maybe, in a future, if we see it as an advantage and not an hindrance" [14:37:05] _joe_: it could be interesting for wikidata / wikibase as it would allow us to try out for example building a "thing" in front of mediawiki, or other such things, that may at least initially only make sense for wikidata [14:37:39] but that would also come with a bunch of other things required, such as, no longer doing direct DB access from wikipedias -> wikidata, instead doing that through apis [14:37:58] (all just a vauge dream) [14:39:01] _joe_ wkandek and elukey, when you have time, I would appreciate your input for T272085 [14:40:03] <_joe_> addshore: doing that access through apis would create a lot of headaches I think :) [14:40:58] meanwhile, _joe_ I think I should remove mc1024 from the cluster and take the resharding hit [14:41:21] even though the gutterpool is working and it is sufficient [14:41:48] mcrouter is probably storing the delete log in each mw server [14:42:00] which is again not terrible, but not useful either [14:42:29] we also need to reboot the mc-gp* hosts for the uprgades [14:44:21] <_joe_> effie: yeah, you know I agree :) [14:44:36] ok, then I will go for it now [14:45:00] do you remember what was the process last time? let puppet take its course [14:45:13] or force puppet to run sooner on mw* ? [14:46:05] I was hoping that we would move forward with the mc* refresh sooner [14:46:12] but I don't think it is possible now [14:46:22] <_joe_> let me read the task [14:46:30] sure [14:47:41] if the specs are the ones that serviceops agreed on then it is ok for me too :) [14:48:02] there is almost zero chance that we'll get 10g links in eqiad though [14:48:10] oh why? [14:48:12] for 18 nodes I mean [14:48:14] ah [14:48:32] well, we will add the 10G NICs anyway [14:48:40] sure makes sense [14:49:21] for the last batch of hadoop workers (24) it has been a problem finding space, I had to work with a lot of people to move servers around, and I am still working with dcops for the last 6 nodes [14:49:41] this assuming that we want to avoid 4 shards on the same rack, or more :) [14:49:45] <_joe_> elukey: as I repeatedly stated, memcached's network capabilities have been our biggest scalability bottleneck for a long time. So I'd assume they're quite high-priority [14:50:05] we have the onhost memcached (hopefully) fully working by the end of this Q [14:50:14] _joe_ you know more than me that I agree, I am not saying the contrary, but there is simply no space atm [14:50:14] <_joe_> that's a bandaid though [14:50:20] yes it is [14:50:20] <_joe_> a clever one [14:50:31] <_joe_> anyways, lemme comment on the task and [14:50:42] <_joe_> regarding the resharding [14:50:53] also, some days ago we had a nasty issue moving data between two hadoop clusters, namely saturating the links router<->switches [14:50:57] <_joe_> do you want the SRE answer or the MW developer answer? [14:51:05] for resharding ? [14:51:26] I want both answers, but wI think we should go on with it regardless [14:51:26] there is a plan to upgrade those links to 100G buut 18 mc nodes can cause some trouble if they start using a ton of bw [14:51:29] it is either that [14:51:30] or [14:51:42] cripple the mc-gp pool by 1 server [14:51:48] I am saying that since bw saturation can affect all production of course [14:51:54] and replace 1024's place in the cluster [14:52:00] that has a smaller impact [14:52:40] having 2 servers for teh gutter pool instead of 3 is fine [14:52:42] effie: the idea that we had for mc-gp was that a single node was beefy enought to tolerate the failure of multiple shards, so I wouldn't reduce the gutter pool size [14:53:00] elukey: if resharding hurts us [14:53:07] <_joe_> so, SRE answer: run puppet on a few servers first, so that only a part of the traffic will get the hit [14:53:09] it is a better bandaid [14:53:11] we have some racks with 3/4 mc shards each, if we get a network failure we are in trouble (say a switch dies) [14:53:43] <_joe_> MW dev answer: are you kidding? the switch needs to be as atomic as possible, because we expect memecache to be a consistent datastore [14:54:11] SRE: it is not supposed to be super consistent, which is why it is a cache and not a db [14:54:30] elukey: that is true [14:54:32] ok [14:54:34] resharding it is [14:56:56] <_joe_> so, if we really have no 10g ports, an alternative approach could be to add shards [14:57:20] <_joe_> we buy el cheapo servers with 64 GB of ram [14:57:31] <_joe_> the smallest spinning disks we can find [14:57:39] <_joe_> and 1 g nics [14:57:43] except that rack space is a bottleneck right now, right? [14:57:46] <_joe_> but we buy 32 of them [14:58:17] 64 is small I think [14:58:23] <_joe_> rzl: I'm trying to propose an alternative approach, but yes it feels like a catch-22 [14:58:31] <_joe_> effie: not if we have 32 shards, no [14:59:26] <_joe_> lemme comment further on the task [14:59:28] and then more connections to various shards to get data etc [14:59:46] I would rather move some servers from one rack to another [15:00:00] in eqiad, in codfw they are nicely distributed [15:00:29] I don't agree, within the next 5 years [15:00:44] we may be able to fully support 10G for all mc servers [15:08:49] 10serviceops, 10Dumps-Generation, 10Platform Engineering, 10SRE, 10Patch-For-Review: Upgrade snapshot hosts to Buster - https://phabricator.wikimedia.org/T269377 (10ArielGlenn) snapshot1010 is done. I need to do a bunch more testing before I can reimage snapshot1008. [15:09:25] _joe_: doing direct db access causes a lot of headaches too, just different ones ;) [15:09:42] <_joe_> addshore: to *you*, not to *me* [15:09:50] xD [15:09:54] <_joe_> that was the whole point, really. [15:10:11] <_joe_> jokes aside, using the db as an integration point is terrible [15:10:31] <_joe_> and I think we should have better ways to treat wikidata dispatching and access to its entities [15:11:03] given the book club im currently in, maybe kafka is the whole answer [15:12:12] <_joe_> when all you have is a hammer... [15:12:42] <_joe_> and btw, you are already using it [15:12:58] indeed, and indeed [15:13:12] <_joe_> that's what I find amusing about all this. We already have an events based architecture, just one that makes sense in practice [15:13:46] <_joe_> but coming back to wikidata, if we could detach wikibase completely from mediawiki, I guess we'd need optimized data access paths for *synchrosous* requests [15:13:59] and the whole client dispatching stuff is also just event based. Data though doesnt come from the event, but from another query then [15:14:26] <_joe_> addshore: just remember, the capital sin of any event-driven architecture is when you need the async bus in the middle of serving a request to the client [15:14:36] <_joe_> when that happens, it's game over, you've overdone it [15:14:48] <_joe_> but back to my point [15:14:58] <_joe_> if wikibase was completely detached from mediawiki [15:15:12] <_joe_> we'd need need optimized query paths [15:15:26] <_joe_> and probably more resources than what we use today [15:15:46] <_joe_> and the ability for mediawiki to fail open if wikibase is unavailable [15:16:22] <_joe_> there are a series of other problems though, like... how do you manage authn/authz? [15:18:13] more resources agreed, though with our new pending creation APIs, where those resources would be (cache vs app) would be an interesting point [15:18:47] as for failure cases, really we should already faily nicely there when s8 falls over (we don't everywhere though), but its the same case [15:19:02] but yes, i agree, many things to think about [15:21:12] <_joe_> heh indeed [15:26:53] <_joe_> serviceopsen: any message to rely to PET? [15:38:32] not I [16:04:33] _joe_: addshore i don't know nuthin about detaching wikibase from mediawiki, but i think https://phabricator.wikimedia.org/T120242 (with possibly putting the wikidata content in events) would allow us to try new things (at least for read models, e.g. caches and wdqs, etc.) [17:27:11] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:32:17] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:33:55] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [17:42:41] <_joe_> ottomata: I deeply dislike the idea of coupling event production to a binlog of a database [17:43:02] <_joe_> for many reasons I won't elaborate on now as I need to go afk :) [17:43:10] <_joe_> but let's find the time to talk about it [17:44:47] _joe_: would love to [18:09:53] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1401.eqiad.wmnet'] ` an... [18:16:49] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2278.codfw.wmnet'] ` an... [18:43:15] 10serviceops, 10SRE, 10User-jijiki: ifup@eno1.service fails on mc* hosts after 4.19.171-2 upgrade - https://phabricator.wikimedia.org/T273918 (10jijiki) [18:48:41] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1309.eqiad.wmnet'] ` an... [19:01:29] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [19:52:13] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:53:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:54:35] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:24:30] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2244.codfw.wmnet'] ` an... [20:33:04] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2267.codfw.wmnet'] ` an... [20:35:28] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1400.eqiad.wmnet'] ` an... [21:02:37] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:04:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:05:24] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:10:43] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1308.eqiad.wmnet'] ` an... [21:21:34] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:28:11] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata) Added some docs here: https://wikitech.wikimedia.org/wiki/Event_P... [21:28:29] 10serviceops, 10Analytics, 10Analytics-Kanban, 10Event-Platform, and 5 others: Set up internal eventstreams instance exposing all streams declared in stream config (and in kafka jumbo) - https://phabricator.wikimedia.org/T269160 (10Ottomata) Yeehaw thank you Luca! [21:46:01] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1399.eqiad.wmnet'] ` an... [21:47:22] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1398.eqiad.wmnet'] ` an... [22:10:56] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:11:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [22:22:14] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1263.eqiad.wmnet'] ` an... [22:38:09] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1311.eqiad.wmnet'] ` an... [22:52:50] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1397.eqiad.wmnet'] ` an... [22:53:57] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1396.eqiad.wmnet'] ` an... [22:54:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq...