[14:12:12] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10kchapman) [14:12:57] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10kchapman) [14:13:32] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10EvanProdromou) Is this something that could be managed with k8s? https://kubernetes.io/docs/tasks/job/automat... [14:13:48] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10kchapman) [14:28:25] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10aezell) @EvanProdromou I think that a solution like that is something closer to what @Joe was proposing than... [14:34:44] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Ottomata) FWIW, Analytics will (eventually?) be considering replacing our scheduler (Oozie) with Apache Airfl... [14:51:16] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10akosiaris) >>! In T218268#5049090, @Ottomata wrote: > > `apt-get update` couldn't connect to the apt source IPv6 addresse... [14:53:44] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) OF COURSE! Thank you Alex. Since (IIUC), the preferred default is that all servers get IPv6 addies, we should... [15:08:15] <_joe_> sigh that was it? [15:08:17] <_joe_> ahahah [15:13:39] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10daniel) Wikidata also has a need for delayed and/or periodic tasks for change dispatching. That is currently... [15:13:56] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10daniel) [15:15:07] did we ever evaluate podman as an alternative to docker? [15:15:33] hears it doesnt have to run as root [15:17:39] <_joe_> mutante: uhm no? [15:17:54] podman is project atomic [15:18:05] which is effectively dead since the acquisition of coreos [15:20:20] ah, i was just talking to a friend about docker and k8s and podman came up because he was looking for something that doesn't require the docker daemon running as root [15:20:34] also note that we don't particularly care about docker itself. If a runtime engine passes the tests that kubernetes has, we may very well consider it [15:20:51] cri-o is one such runtime engine (and it's being pushed heavily) [15:20:56] <_joe_> we care about the dockerfile format and docker images and the rest of the ecosystem [15:20:57] rkt on the other hand... failed [15:21:10] _joe_: that's OCI [15:21:14] <_joe_> not about the runtime [15:21:27] already docker independent [15:21:44] <_joe_> yeah, it's OCI because docker created them, but sure, that's what I was saying [15:22:03] https://kubernetes.io/docs/setup/cri/ fwiw [15:23:18] <_joe_> and yes, cri-o is interesting in some ways [15:23:35] ottomata: I 've enabled https://gerrit.wikimedia.org/r/499223 on the staging cluster. Wanna try and see if it solves the issue? At least to have a verification it was to blame [15:24:08] ok! just gotta undo the prefer ipv4 thing [15:25:40] oo actually lets make it prefer ipv6 and see for sure [15:33:06] _joe_: T219279 is… fun. Blocker to wider PHP7 testing? [15:33:20] hm i have a template problem. akosiaris is there a way to see a rendered template without logging into the container? [15:33:59] oh oh oh i think i see it maybe. i can't override an existing config? [15:33:59] hm [15:34:05] helm template [15:34:08] duplicated mapping key [15:34:26] there is also helm upgrade --dry-run --debug [15:34:29] <_joe_> James_F: looks like it, we might want people in #mediawiki-code to take a look [15:34:29] both can help [15:34:33] danke will look [15:34:34] <_joe_> *coire [15:34:36] <_joe_> meh [15:34:42] * _joe_ needs coffee [15:35:59] ah i see. [15:35:59] hm [15:36:22] ok i'm just going to downgrade chart to get the old version, will be easier [15:36:52] 10serviceops, 10Operations, 10Traffic, 10User-jijiki: Allow directing a percentage of API traffic to PHP7 - https://phabricator.wikimedia.org/T219129 (10jijiki) [15:40:30] akosiaris: how can I use an older chart version but with different values? [15:40:37] can I -f values.yaml on rollback command? [15:42:06] ah upgrade --version [15:42:07] ? [15:42:12] yup [15:42:16] gr8 [15:42:30] ottomata: which answers btw the question we had about the old versions :P [15:42:45] it's nice keeping them around :-) [15:42:47] oh to keep them [15:42:47] aye [15:45:44] _joe_: Yeah, I'll flag to people. [15:47:26] akosiaris: kafka-jumbo1003.eqiad.wmnet:9092/bootstrap: Connected to ipv6#[2620:0:861:102:10:64:16:99]:9092 [15:47:27] looks good! [15:47:57] cool [15:47:58] thanks! [15:48:05] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) After ^: ` kafka-jumbo1003.eqiad.wmnet:9092/bootstrap: Connected to ipv6#[2620:0:861:102:10:64:16:99]:9092 ` L... [15:48:08] I 'll merge then and deploy to all envs [15:48:09] do you think I should remove my prefer ipv4 setting? [15:48:20] rdkafka is round-robining whatever it gets from dns [15:48:38] maybe for consistency its best to prefer one or the other? [15:48:54] for consistency & repeatability [15:49:33] difficult to answer that. Depends on how much you want to dive into IPv6 vs IPv4 every time you debug [15:49:53] preferring v4 will also catch some people that don't expect it of guard as well [15:51:09] ottomata: ok, applied everywhere [15:51:30] I did not expect that a IPv4 LVS service would end up causing a IPv6 look up [15:52:06] aye, but the kafka backend isn't lvs [16:33:44] 10serviceops, 10Operations, 10Patch-For-Review: Use PHP7 to run maintenance scripts - https://phabricator.wikimedia.org/T219135 (10Krinkle) >>! In T219135#5054661, @Dzahn wrote: >>>! In T219135#5054561, @Krinkle wrote: >> Duplicate of T195392? > > Yes. Just not sure how to merge them best. Unfortunately Pha... [16:34:18] 10serviceops, 10Operations, 10Patch-For-Review: Use PHP7 to run maintenance scripts - https://phabricator.wikimedia.org/T219135 (10Krinkle) [16:34:31] 10serviceops, 10Operations: SRE FY2019 Q4 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10Krinkle) [16:36:29] 10serviceops, 10Operations, 10Patch-For-Review: Use PHP7 to run maintenance scripts - https://phabricator.wikimedia.org/T219135 (10Krinkle) [17:25:07] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Niharika) >>! In T218812#5058017, @aezell wrote: > @EvanProdromou I think that a solution like that is someth... [17:32:37] akosiaris: ok am stumped. [17:32:37] i [17:32:53] trying to upgrade staging, can't figure out where my values are coming from. [17:33:08] they aren't in values files [17:33:10] they aren't from CLI [17:33:16] they aren't in template. [17:33:34] in COMPUTED VALUES: [17:33:43] e.g. [17:33:47] extra_kafka_conf: [17:33:47] broker.address.family: v6 [17:34:04] do un-overridden values persist between upgrades? [17:37:26] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Mooeypoo) I also think that making sure MediaWiki is installable and usable in shared hosting is a worthy goa... [17:40:35] actually thata is in USER-SUPPLIED VALUES too [17:40:36] BUT where?! [17:40:39] i'm not supplying it anywhere [17:42:13] ah ha [17:42:17] the answer is yes [17:42:22] i needed --reset-values [18:44:09] ottomata: we keep values around by explicitly passing --reuse-values. That's kind of on purpose, but overall if you always pass -f values.yaml we can probably drop it [18:44:20] it's to reevaluate in helmfile that is [18:44:28] I am having some pretty good results with it btw [18:47:22] ok, i think i wasn't expecting that [18:47:44] i'll just note that I want to always use --reset-values with eventgate stuff, otherwises it is a little magical and confusing [18:48:04] i always pass -f values. [18:49:09] <_joe_> come work in serviceops, we have the best bugs! https://phabricator.wikimedia.org/T219279 [18:49:12] <_joe_> #FML [18:50:25] oof [18:50:42] <_joe_> this is actually a php bug that's been around forever and they fixed [18:51:04] <_joe_> and now we're left with multiple pages for the same title [18:53:29] ah welcome (back) to unicode hell [18:57:12] :-) [18:59:04] There's already a maintenance script to fix pages which have now-illegal titles, mostly for when we create new namespaces (Foo:Bar is a fine article title until we create the Foo namespace, after which the NS0 page Foo:Bar can't be accessed). Don't recall if it does Unicode fix-ups too, but we could alter it an run it to rescue impossible pages. The only problem is during cut-over when (e.g.) admins on PHP7 can't even see a page that's [18:59:04] been vandalised. [19:03:03] namespaceDupes [19:03:16] it doesn't seem to do anything with unicode changes [19:04:13] I wonder if cleanupTitles might ... hmmm [19:21:48] Oh, right, we have two similar scripts, they're not one. [19:31:23] what's the best deployment-pipeline docuementation overview page? [19:31:44] https://wikitech.wikimedia.org/wiki/Deployment_pipeline [19:31:45] ? [21:04:20] 10serviceops, 10Analytics, 10ChangeProp, 10Community-Tech, and 6 others: Provide the ability to have time-delayed or time-offset jobs in the job queue - https://phabricator.wikimedia.org/T218812 (10Krinkle) >>! Task description: > If a user adds an expiring item to their watchlist, when the expiration date... [21:04:27] 10serviceops, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Services (watching), 10User-ArielGlenn: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10mobrovac) [21:40:10] 10serviceops, 10Operations, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Doing), and 3 others: Enabling api-request eventgate to group1 caused minor service disruptions - https://phabricator.wikimedia.org/T218255 (10akosiaris) [21:40:22] 10serviceops, 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10akosiaris) 05Open→03Resolved Resolving, feel free to reopen [21:56:30] 10serviceops, 10Analytics, 10EventBus, 10Release Pipeline, 10Services (watching): Modern Event Platform: Stream Intake Service: Documentation - https://phabricator.wikimedia.org/T219332 (10Ottomata) [21:56:35] 10serviceops, 10Analytics, 10EventBus, 10Release Pipeline, 10Services (watching): Modern Event Platform: Stream Intake Service: Documentation - https://phabricator.wikimedia.org/T219332 (10Ottomata) [21:57:41] 10serviceops, 10Analytics, 10EventBus, 10Release Pipeline, 10Services (watching): Modern Event Platform: Stream Intake Service: Documentation - https://phabricator.wikimedia.org/T219332 (10Ottomata) Starting on some great stuff here: https://wikitech.wikimedia.org/wiki/Event_Platform https://wikitech.w...