[05:29:08] 10serviceops, 10MW-on-K8s, 10Operations: Sandbox/limit child processes within a container runtime - https://phabricator.wikimedia.org/T252745 (10Joe) [05:29:47] 10serviceops, 10MW-on-K8s, 10Operations: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10Joe) [07:39:20] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10Joe) [07:39:42] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10ops-monitoring-bot) Icinga downtime for 7 days, 0:00:00 set by jayme@cumin1001 on 3 host(s) and their services w... [07:47:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10JMeybohm) VMs have been shut down as of now. Will decommission on Wednesday if nothing pops up. [09:03:43] Good morning, FYI I've made this change to the LVS wikitech page for the Netbox IP allocation. LMK if you have any question/comments. [09:03:46] https://wikitech.wikimedia.org/w/index.php?title=LVS&type=revision&diff=1881134&oldid=1880104 [09:07:45] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Errors regarding k8s services/nodes in pybal logs - https://phabricator.wikimedia.org/T262802 (10JMeybohm) [12:41:43] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC, 10Patch-For-Review: RFC: PHP microservice for containerized shell execution - https://phabricator.wikimedia.org/T260330 (10tstarling) An open question is what to do about shell pipelines. Currently if you do `Shell::command('foo|bar')` then foo will... [14:28:24] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move mobileapps to use TLS only - https://phabricator.wikimedia.org/T255876 (10JMeybohm) a:03JMeybohm [14:28:48] 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review, 10Release Pipeline (Blubber): Move blubberoid to use TLS only. - https://phabricator.wikimedia.org/T236017 (10JMeybohm) a:05Joe→03JMeybohm [15:02:41] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10Joe) [15:18:45] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Store Kubernetes events for more than one hour - https://phabricator.wikimedia.org/T262675 (10lmata) hello @JMeybohm do you have some guidance as to priority for this task, is it interesting for the next set of weeks? or is this more along the... [15:27:44] 10serviceops, 10MediaWiki-Cache, 10MediaWiki-General, 10Performance-Team, 10User-jijiki: Use monotonic clock instead of microtime() for perf measures in MW PHP - https://phabricator.wikimedia.org/T245464 (10jijiki) [15:57:54] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Store Kubernetes events for more than one hour - https://phabricator.wikimedia.org/T262675 (10JMeybohm) Hi @lmata, I would love to get this done at the beginning of next quarter (mainly because we're probably going to do a lot of kubernetes up... [16:14:09] _joe_: that ATS purge is reverted as discussed 👍 [16:19:24] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Store Kubernetes events for more than one hour - https://phabricator.wikimedia.org/T262675 (10herron) >>! In T262675#6459313, @JMeybohm wrote: > What I think I need from your side is mainly the "okay" to push those events to the `logstash-*` i... [17:43:25] new parsoid servers pooled - CPU/network per host going down on old servers - [17:46:36] mutante: they are all still marked as staged in netbox [17:46:59] volans: oh, fixing ! [17:47:28] thx [19:59:06] 10serviceops, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: [Beta Cluster] How can secrets be stored for use in a docker_services service configuration? - https://phabricator.wikimedia.org/T262552 (10Mholloway) Ping @jijiki and @akosiaris... [20:17:17] 10serviceops, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: [Beta Cluster] How can secrets be stored for use in a docker_services service configuration? - https://phabricator.wikimedia.org/T262552 (10bd808) The only place to store "secrets... [20:50:57] 10serviceops, 10Operations: move 20 new codfw parsoid servers (parse2*) into production - https://phabricator.wikimedia.org/T247441 (10Dzahn) [20:58:57] 10serviceops, 10Operations: move 20 new codfw parsoid servers (parse2*) into production - https://phabricator.wikimedia.org/T247441 (10Dzahn) 05Open→03Resolved - servers had OS installed - servers had puppet role applied - icinga checks confirmed all green - added to conftool data - set weight for all to 1... [21:35:39] it occurs to me that I don't actually know how to perform a rolling restart of a k8s service [21:36:20] it also occurs to me that perhaps 5:35pm local time might not be the ideal time to experiment [21:40:26] haha I'm just checking out for the day so I can't disagree [21:40:28] good luck! [21:40:55] cool cool, the thing I want to do is in kubectl 1.15, but we run 1.13 [21:54:28] so what I see is, a ReplicaSet that says: [21:54:30] Replicas: 1 current / 1 desired [21:54:32] Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed [21:54:58] and also, I see in `kubectl get pods` output: eventgate-logging-external-production-79f8b8bc48-dvpnh 3/3 Running 0 5d6h [21:55:28] and I don't see where that 3 comes from except if we manually scaled it, in which case I have no idea if the replicaset is controlling it anymore