[01:19:53] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Jdforrester-WMF) The idea of using mtime was always to avoid having to inspect the contents of the IS file itself. M... [03:38:08] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Krinkle) If I understand @thcipriani 's conclusion correctly, the problem is that code execution from disk (opcache)... [08:55:12] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Joe) >>! In T236104#5914485, @Krinkle wrote: > If I understand @thcipriani 's conclusion correctly, the problem is t... [09:01:10] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Joe) I think we need to go at this another way. We should refresh the json cache only when `filemtime('InitialiseSet... [10:01:38] 10serviceops: Envoy can't connect to servers using TLS 1.3 (but can serve TLS 1.3 to clients) - https://phabricator.wikimedia.org/T246083 (10Joe) [10:09:20] _joe_: when you are around, can we discuss last details for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/569541/ [10:09:34] it is not changing anything on the current config anyway [11:05:40] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10jijiki) If there are no objections, I would like to proceed with this [14:00:41] 2 increases in appserver latency, at 1:00 and another at 13:30 [14:34:29] oh boy [14:34:30] :) [14:45:58] woohoo [14:58:46] 10serviceops, 10Beta-Cluster-Infrastructure, 10Operations, 10observability: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) After a lot of fiddling with @herron, we are finally at this https://phabricator.wikimedia.org/P10513 ! The resource field... [15:20:14] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar), 10Wikimedia-Incident: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Lucas_Werkmeister_WMDE) Adding #wikimedia-incident for [20200225-mediawiki interface languag... [15:41:02] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar), 10Wikimedia-Incident: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Krinkle) >>! In T236104#5914734, @Joe wrote: >>>! In T236104#5914485, @Krinkle wrote: >> * I... [15:46:02] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar), 10Wikimedia-Incident: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Krinkle) >>! In T236104#5914744, @Joe wrote: > We should refresh the json cache only when `f... [16:00:56] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar), 10Wikimedia-Incident: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Joe) >>! In T236104#5916074, @Krinkle wrote: >>>! In T236104#5914734, @Joe wrote: >>>>! In T... [16:22:30] 10serviceops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need by: 2020-02-12) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Cmjohnson) [17:49:24] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10WDoranWMF) Moving this to feature requests for PMs to review, we'll need to investigate what appropriate limits woul... [17:57:25] 10serviceops, 10MediaWiki-Cache, 10MediaWiki-General, 10Performance-Team: Use monotonic clock instead of microtime() for perf measures in MW PHP - https://phabricator.wikimedia.org/T245464 (10Krinkle) The main problem I'd like to solve is to remove some doubt from our various backend timing instrumentation... [18:00:07] 10serviceops, 10Core Platform Team, 10MediaWiki-General, 10Operations, 10Wikimedia-Incident: Revisit timeouts, concurrency limits in remote HTTP calls from MediaWiki - https://phabricator.wikimedia.org/T245170 (10Krinkle) [19:15:19] q, is there no api.svc.codfw? [19:15:53] OHthere is [19:15:53] hm [19:15:57] it hung for asec [19:15:58] nm [19:23:12] 10serviceops, 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar), 10Wikimedia-Incident: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10thcipriani) Example code above for anyone interested in fiddling with this: https://github.c... [19:47:36] 10serviceops, 10Deployments, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10greg) [20:18:59] 10serviceops, 10Operations, 10Performance-Team, 10Patch-For-Review: Test gutter pool failover in production and memcached 1.5.x - https://phabricator.wikimedia.org/T240684 (10jijiki) @aaron @Krinkle Please let us know if the configuration and the information on this task are enough to proceed with testing... [20:51:52] 10serviceops, 10WMF-JobQueue, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: Jobrunner monitoring still calles /rpc/runJobs.php - https://phabricator.wikimedia.org/T243096 (10Pchelolo) a:03hnowlan Per conversation with @Joe regarding the future of this monitoring check, all we ne... [21:39:43] 10serviceops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10wiki_willy) [21:59:46] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Dzahn) [22:02:24] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Dzahn) Hi all, those of you who have not signed it yet, please read and sign L3. @Sbailey Please create a new SSH keypair a... [22:03:30] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Dzahn) @ssastry @Jdforrester-WMF Should sbailey be added to the parsoid-admins group as well so that all people who can deploy... [23:54:34] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10ssastry) >>! In T245877#5917965, @Dzahn wrote: > > those of you who have not signed it yet, please read and sign L3. Looks l... [23:58:03] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Dzahn) @ssastry Yea, though it's not tied to the hostname. It's "the puppet role parsoid::testing installs the admin groups...