[01:25:15] <wikibugs>	 10serviceops, 10Operations, 10Phabricator: Puppet using the phabricator class fails with: "secret(): invalid secret waf/modsecurity_admin.conf" - https://phabricator.wikimedia.org/T221182 (10Paladox)
[01:32:36] <wikibugs>	 10serviceops, 10Operations, 10Phabricator: Puppet using the phabricator class fails with: "secret(): invalid secret waf/modsecurity_admin.conf" - https://phabricator.wikimedia.org/T221182 (10Paladox) 05Open→03Declined Oh, nvm, labs/private was out of date too.
[08:42:35] <wikibugs>	 10serviceops, 10Operations, 10Thumbor: Export useful metrics from haproxy logs for Thumbor - https://phabricator.wikimedia.org/T220499 (10Gilles) a:05Gilles→03jijiki
[08:45:03] <wikibugs>	 10serviceops, 10Operations, 10Thumbor: Export useful metrics from haproxy logs for Thumbor - https://phabricator.wikimedia.org/T220499 (10Gilles) I'm not seeing the metrics show up in the "eqiad/prometheus" ops datasource in Grafana. I'm not sure how prometheus is supposed to be configured to collect the dat...
[13:29:48] <ottomata>	 akosiaris:  is a 300mb limit ok?
[13:29:52] <ottomata>	 150 requests 300mb?
[13:30:12] <ottomata>	 figured if 100-150 is the avg, something higher than 200 might be better for a limit
[13:30:21] <akosiaris>	 I think not, lemme make sure
[13:31:19] <ottomata>	 ok
[13:31:28] <akosiaris>	 hmmm
[13:31:28] <ottomata>	 the worker heap limit is set at 200mb
[13:31:41] <ottomata>	 i guess the master restarts the worker if it tries to allocate more than that on heap
[13:31:43] <akosiaris>	 so either my grafana math is wrong, or the top pods in https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate?refresh=1m&panelId=31&fullscreen&edit&orgId=1&from=now-7d&to=now
[13:31:46] <akosiaris>	 are at 630MB
[13:32:31] <ottomata>	 oh that's a lot....and those haven't been under load...
[13:32:42] <ottomata>	 but, are they alllowed to use that much?
[13:32:57] <ottomata>	 i guesss thata includes the statsd exporter?
[13:33:20] <akosiaris>	 no that's topk(5, service_runner_rss_heap_kilobytes_sum{job="k8s-pods", service="$service", kubernetes_namespace="$service"})
[13:33:29] <akosiaris>	 so it's just nodejs
[13:33:39] <ottomata>	 ah
[13:33:48] <akosiaris>	 that being said, that adds not seems to match the metrics reported from cadvisor
[13:34:14] <akosiaris>	 container_memory_usage_bytes{namespace="eventgate-analytics"} reports indeed stuff around 175MB
[13:34:18] <ottomata>	 where's it getting the the MB unit?
[13:35:49] <akosiaris>	 yeah avg(container_memory_usage_bytes{namespace="eventgate-analytics", container_name="eventgate-analytics-production"}) on eqiad prometheus/k8s says 120MiB 
[13:35:53] <akosiaris>	 hmmm
[13:35:59] <akosiaris>	 what are these other numbers then 
[13:36:30] <akosiaris>	 service runner miscalculating? or me having used the wrong formula
[13:38:26] <akosiaris>	 hmm possibly the latter. I see service_runner_rss_heap_kilobytes_sum{service="$service"}/service_runner_rss_heap_kilobytes_count{service="$service"} returning more like 55MB
[13:38:39] <akosiaris>	 which sounds more reasonable
[13:40:03] <ottomata>	 hm, whata about _bucket with le="Inf"
[13:40:08] <ottomata>	 topk(5, service_runner_rss_heap_kilobytes_bucket{service="eventgate-analytics",le="+Inf"})
[13:40:30] <ottomata>	 oh no that is like count
[13:40:31] <ottomata>	 sory
[13:42:20] <akosiaris>	 ok found it
[13:42:24] <akosiaris>	 wrong formula on my part
[13:42:39] <akosiaris>	 wrong unit and wrong formula in fact
[13:42:43] <ottomata>	 they / count?
[13:42:44] <ottomata>	 the*
[13:42:58] <akosiaris>	 yeah the correct is topk(5, service_runner_rss_heap_kilobytes_sum{job="k8s-pods", service="$service", kubernetes_namespace="$service"}/service_runner_rss_heap_kilobytes_count{job="k8s-pods", service="$service", kubernetes_namespace="$service"})
[13:43:20] <akosiaris>	 so avg of 55MB ? knock yourself out with 300MB
[13:43:22] <akosiaris>	 fine by me
[13:43:42] * akosiaris fixing a lot of formulas now
[13:44:00] <akosiaris>	 at least it's copy/paste
[13:44:13] <ottomata>	 hah ok
[13:44:19] <ottomata>	 well prod is unloaded
[13:44:34] <ottomata>	 max in staging from yesterdaay might be a betterr thing to look at
[13:44:57] <ottomata>	 but i still only see 80MB there
[13:45:01] <ottomata>	 oh no
[13:45:04] <ottomata>	 that is with the bad formulat
[13:45:09] <ottomata>	 ok, 150 300
[13:45:12] <ottomata>	 i like it
[13:45:45] <ottomata>	 akosiaris why do we do histogram for memory usage? 
[13:45:54] <ottomata>	 don't we just want the current value reported?
[13:46:20] <akosiaris>	 for service_runner_rss_heap_kilobytes_sum you mean?
[13:46:38] <akosiaris>	 it's not a histogram, it's a summary. And the only reason is nodejs/service-runner
[13:46:51] <akosiaris>	 it should be a gauge but service-runner abuses a timer for that
[13:47:33] <akosiaris>	 ottomata: https://github.com/wikimedia/operations-deployment-charts/blob/master/charts/mathoid/config/prometheus-statsd.conf#L13
[13:47:35] <ottomata>	 i mean  aprometheus historgram type
[13:47:37] <ottomata>	   name: service_runner_${2}_heap_kilobytes
[13:47:37] <ottomata>	     timer_type: histogram
[13:47:53] <ottomata>	 but ok 'abuse' is the reasaon :p
[13:48:55] <akosiaris>	 and it's not a summary, it's a histogram you are correct
[13:49:04] <akosiaris>	 I got carried away by the _sum
[13:49:13] <akosiaris>	 but that exists in both summaries AND histograms
[13:52:58] <ottomata>	 aye
[13:55:00] <akosiaris>	 ok, all dashboards updated
[13:55:03] <akosiaris>	 nice catch 
[13:56:17] <ottomata>	 thnaks
[13:57:40] <ottomata>	 akosiaris:  also there's this
[13:57:41] <ottomata>	 https://phabricator.wikimedia.org/T220709
[13:57:49] <ottomata>	 which i guess you saw
[13:57:51] <akosiaris>	 yup, known
[13:57:58] <akosiaris>	 I am waiting on it too, to update the images
[13:58:15] <akosiaris>	 I have the same problem as you do with the mathoid+cxserver deployments
[13:58:33] <akosiaris>	 gc stats are calculated wrong
[13:59:11] <akosiaris>	 https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate?refresh=1m&panelId=25&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fk8s-staging&var-service=eventgate-analytics&var-kafka_topic=All&var-kafka_broker=All&var-kafka_producer_type=All&from=now-7d&to=now
[13:59:22] <akosiaris>	 heh.. even that points out some 80MB under your tests
[13:59:36] <akosiaris>	 nice
[13:59:55] <ottomata>	 hm akosiaris  so maybe the mem limits should actually be lower?
[13:59:57] <akosiaris>	 with spikes however up to 160
[14:00:11] <ottomata>	 were your comments on the task based on the wrong metrics?
[14:00:15] <ottomata>	 _sum?
[14:00:26] <akosiaris>	 no I was looking at the correct ones, albeit taking the max()
[14:00:31] <ottomata>	 ok
[14:00:49] <akosiaris>	 that's btw the sum ^
[14:01:00] <ottomata>	 sum/count ya?
[14:01:06] <akosiaris>	 in production in the same graph you will see something like 1GB
[14:01:10] <akosiaris>	 that's across all pods
[14:01:12] <ottomata>	 oh
[14:01:13] <ottomata>	 k
[14:01:14] <akosiaris>	 that might not make sense
[14:01:22] <akosiaris>	 but we can always amend
[14:01:46] <ottomata>	 its possible in prod there will be more mem usage, there will be more topics, and iirc produce batches are done per topic
[14:02:18] <ottomata>	 if you are ok we can leave 150/300 for now?
[14:02:46] <akosiaris>	 sure
[14:02:59] <ottomata>	 k
[14:03:32] <akosiaris>	 cpu wise make sure to use the 200m for requests though. We don't want to tell the scheduler to dedicate an entire CPU to each pod
[14:05:05] <akosiaris>	 limits wise, I am impressed btw that we did hit the 1000m limit. This is not depicted in prometheus in https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate?refresh=1m&panelId=28&fullscreen&orgId=1&from=now-2d&to=now&var-dc=eqiad%20prometheus%2Fk8s-staging&var-service=eventgate-analytics&var-kafka_topic=All&var-kafka_broker=All&var-kafka_producer_type=All
[14:06:04] <ottomata>	 hm
[14:06:23] <ottomata>	 so, requests is just for pod placement, the limit is the real limit value?
[14:06:29] <akosiaris>	 yes
[14:06:31] <ottomata>	 k8s doesn't e.g. expand CPU for a pod 
[14:06:50] <akosiaris>	 no, if a container hits the CPU limit it gets throttled
[14:07:04] <ottomata>	 hm these are sampled over 5ms
[14:07:14] <akosiaris>	 it's of course not that clear cut as accounting for CPU is not that easy
[14:07:16] <ottomata>	 my ab tests doen't last that long
[14:07:37] <akosiaris>	 even over 2m there isn't any substantial diff
[14:07:38] <ottomata>	 i'll try to run one for 10 mins on staging and see how it does
[14:07:54] <akosiaris>	 or using irate() instead of rate()
[14:08:40] <ottomata>	 running for 5mins*
[14:08:41] <ottomata>	 aye
[14:14:41] <ottomata>	 getting up there
[14:14:42] <ottomata>	 https://grafana.wikimedia.org/d/ePFPOkqiz/eventgate?refresh=1m&orgId=1&from=1555508675876&to=1555510475876&var-dc=eqiad%20prometheus%2Fk8s-staging&var-service=eventgate-analytics&var-kafka_topic=All&var-kafka_broker=All&var-kafka_producer_type=All&edit
[14:14:54] <akosiaris>	 you do know you ask all the right questions, right ?
[14:15:04] <akosiaris>	 I am TILing right now
[14:15:17] <ottomata>	 haha that's good to know!
[14:15:25] <ottomata>	 no idea if they are rright when I ask then :p
[14:15:34] <ottomata>	 TIL i ask the right questions
[14:18:41] <akosiaris>	 irate(container_cpu_cfs_throttled_seconds_total{job="k8s-node-cadvisor", namespace="$service", pod_name=~"$service.*", container_name=~"$service.*"}[2m]) says eventgate-analytics gets throttled quite often in eqiad
[14:18:49] <akosiaris>	 some 10ms on avg() everytime
[14:19:02] <akosiaris>	 so you definitely need higher cpu limits stanza
[14:19:56] <akosiaris>	 the max was on 2019-04-05 with 262ms of throttling
[14:20:08] <fsero>	 re https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-manager/
[14:21:36] <ottomata>	 oo that is a nice one
[14:21:38] <akosiaris>	 hm, interesting read
[14:21:55] <ottomata>	 akosiaris:  would the throttling possibly be the reason for the missed heartbeat problem?
[14:22:00] <akosiaris>	 so we should use that to dedicate entire cpus to pods
[14:22:07] <akosiaris>	 ottomata: yeah definitely could be
[14:22:16] <ottomata>	 that would make so much sense
[14:22:26] <akosiaris>	 although the stats are small 
[14:22:32] <ottomata>	 i couldn't figure out what was actually blocking the cpu in the straces or in the flamegraphs
[14:22:37] <akosiaris>	 20ms of throttling should not so easily account for 7.5s 
[14:22:40] <ottomata>	 not that I knew how to read them well
[14:22:42] <ottomata>	 hmm 
[14:22:43] <ottomata>	 oh
[14:23:06] <akosiaris>	 on the other hand... these are not exactly up to the second stats
[14:23:25] <akosiaris>	 but still... we are talking 3 orders of magnitude here
[14:24:39] <ottomata>	 akosiaris:  it looks like at 2.2K reqs/second (with a single topic and a pretty simple json message to validate), staging pod levels off at 1.2 cpu seconds
[14:25:25] <ottomata>	 so i think our limit of 2 cores is good
[14:25:52] <ottomata>	 in prod each pod will probably average 400 reqs per second
[14:26:02] <akosiaris>	 I see 1.5s 
[14:26:05] <ottomata>	 max average
[14:26:06] <ottomata>	 oh nicie
[14:26:11] <akosiaris>	 max() though
[14:26:13] <akosiaris>	 not avg()
[14:26:16] <ottomata>	 you do?
[14:26:18] <ottomata>	 oh max
[14:26:24] <ottomata>	 ok cool, 'im just looking at the one in the dash
[14:26:28] <akosiaris>	 max(irate(container_cpu_usage_seconds_total{job="k8s-node-cadvisor", namespace="$service", pod_name=~"$service.*", container_name=~"$service.*"}[2m]))
[14:28:17] <akosiaris>	 ah... container_cpu_usage_seconds_total vs container_cpu_user_seconds_total
[14:28:26] <akosiaris>	 the former has system + user in it 
[14:28:51] <akosiaris>	 probably also IOwait (although this here is irrelevant, this doesn't write anything to disk)
[14:29:41] <akosiaris>	 ottomata: anyway 2000m sounds good to me
[14:30:07] <ottomata>	 aye
[14:30:09] <fsero>	 ottomata: in case of still hitting this i recommend you to set the same resources for requests and limits
[14:30:15] <fsero>	 making it Guaranteed
[14:30:29] <fsero>	 however dont do that by default
[14:30:36] <fsero>	 only when you have proved that is still working
[14:30:40] <fsero>	 *not working
[14:30:56] <ottomata>	 hm ok, we might indeed want to bump up requests a little bit mor than 200Mi, we'll see how iti looks in prod under load i think
[14:31:09] <ottomata>	 but 2000Mi sould sholud be a good limit
[14:31:16] <akosiaris>	 requests? you mean limits ? :P
[14:31:31] <ottomata>	 s/Mi/m/
[14:31:32] <akosiaris>	 isn't it nicely confusing? :-)
[14:31:51] <ottomata>	 no fsero  was recommending to possible bump up requests
[14:32:01] <ottomata>	 currently requests: 200m limit:2000m
[14:32:12] <ottomata>	 depending on how it looks in prod
[14:32:17] <fsero>	 https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed
[14:32:19] <akosiaris>	 ah, in order to make the allocation guaranteed
[14:32:28] <fsero>	 there are three kinds of pods attending to quality of seervice
[14:32:46] <fsero>	 if you want a pod to be guaranteed resources assigned to request should be the same ones on limits
[14:38:55] <ottomata>	 ok prod updated with new limits, gonna run the same ab there
[14:49:03] <ottomata>	 ok cool, i ran two abs from two different machines with -c1000
[14:49:36] <ottomata>	 they eventually both died due to too socket timeout/too many open sockets, but I was able to push prod to 26K msgs/second
[14:50:01] <ottomata>	 with CPU max ~1.2s
[14:50:17] <ottomata>	 the currently max planned usage is around 10K msgs/second
[14:50:23] <ottomata>	 so  think we good!
[14:52:44] <akosiaris>	 cool
[19:14:10] <wikibugs>	 10serviceops, 10Core Platform Team Backlog, 10MediaWiki-General-or-Unknown, 10Operations, and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10kchapman) >>! In T219279#5096304, @Joe wrote: >>>! In T219279#5095261...
[19:16:26] <wikibugs>	 10serviceops, 10Core Platform Team Kanban, 10MediaWiki-General-or-Unknown, 10Operations, and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10kchapman)
[19:17:03] <wikibugs>	 10serviceops, 10MediaWiki-General-or-Unknown, 10Operations, 10Core Platform Team (PHP7 (TEC4)), and 4 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10kchapman)
[19:25:00] <wikibugs>	 10serviceops, 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10Core Platform Team Backlog (Later), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Is this related to T218609 and T200832?
[21:30:27] <mutante>	 _joe_: https://gerrit.wikimedia.org/r/c/operations/puppet/+/504791
[21:30:35] <mutante>	 (it doesnt have to be now.. )
[21:30:42] <wikibugs>	 10serviceops, 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10Core Platform Team Backlog (Later), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) It's not just going to become a problem once T198901 is done, it's already a problem -...
[21:33:58] <mutante>	 and another one  https://gerrit.wikimedia.org/r/504793
[21:49:21] <mutante>	 finally https://gerrit.wikimedia.org/r/c/operations/puppet/+/504794
[22:00:50] <wikibugs>	 10serviceops, 10Operations, 10hardware-requests: requesting WMF7426 as phabricator system in eqiad - https://phabricator.wikimedia.org/T215335 (10Dzahn)
[22:02:23] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10Dzahn)
[22:02:33] <wikibugs>	 10serviceops, 10Operations, 10Phabricator, 10Patch-For-Review, 10Release-Engineering-Team (Watching / External): Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Dzahn) 05Open→03Stalled blocked on T215335
[22:05:16] <wikibugs>	 10serviceops, 10Operations, 10hardware-requests: requesting WMF7426 as phabricator system in eqiad - https://phabricator.wikimedia.org/T215335 (10Dzahn) @Robh Now that you are back.. can i please have this server assigned to me? That would unblock T190568 which has been waiting for quite a bit.  It has alrea...
[22:52:34] <wikibugs>	 10serviceops, 10Discovery-Search, 10Elasticsearch, 10RESTBase-Cassandra, and 3 others: Determine future of bare-metal hosting for services like WDQS, ElasticSearch, RESTBase Cassandra, etc. - https://phabricator.wikimedia.org/T221315 (10Jdforrester-WMF)
[22:55:34] <wikibugs>	 10serviceops, 10Discovery-Search, 10Elasticsearch, 10RESTBase-Cassandra, and 3 others: Determine future of bare-metal hosting for services like WDQS, ElasticSearch, RESTBase Cassandra, etc. - https://phabricator.wikimedia.org/T221315 (10Smalyshev) The web UI frontend is just a bunch of HTML files, so there...