[00:29:53] 10serviceops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need by: 2020-02-28) rack/setup/install mw[1385-1413].eqiad.wmnet - https://phabricator.wikimedia.org/T241849 (10Jclark-ctr) @Cmjohnson Host are racked have started power cables Will have cables finished tomorrow name rack_name position mw13... [01:51:24] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10wiki_willy) [08:51:54] 10serviceops, 10Deployments, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Joe) I just made a tiny change to your test.php file https://github.com/lavage... [09:24:16] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10akosiaris) Fine by me. [09:41:29] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10Joe) 05Open→03Declined No please. I didn't notice this task. Scap proxies are under a ton of pressure during train releases, ti was a deliberate choice not to mix the... [10:46:34] I cannot remember who was working on this, but I know it was someone from service ops [10:47:23] Could I replace the useless "disk utilization" graph on the host overview grafana dashboard [10:47:38] with an actual useful disk utilization graph, e.g. IOPS [10:52:24] <_joe_> I think you remember incorrectly [10:52:48] <_joe_> and this should be asked in #sre anyways, to make sure to reach everyone [11:06:08] sorry, wrong channel, I meant to write in foundation (observability) [13:33:49] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10jijiki) 05Declined→03Open (Reopening to discuss this a bit more) I understand your point, automating would be great, but it will not solve the problem where we have ab... [14:08:49] 10serviceops, 10Operations, 10Release-Engineering-Team: mcrouter proxies and scap proxies - https://phabricator.wikimedia.org/T245841 (10Joe) >>! In T245841#5919608, @jijiki wrote: > (Reopening to discuss this a bit more) > > I understand your point, automating would be great, but it will not solve the prob... [14:12:25] 10serviceops, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by volans@cumin2001 for hosts: `kubetcd2001.codfw.wmnet` - kubetcd2001.codfw.wmnet (**... [14:21:00] 10serviceops, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by volans@cumin2001 for hosts: `kubetcd2001.codfw.wmnet` - kubetcd2001.codfw.wmnet (**... [15:19:09] 10serviceops, 10Patch-For-Review: setup new, buster based, kubernetes etcd servers for staging/codfw/eqiad cluster - https://phabricator.wikimedia.org/T239835 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: `kubetcd[2002-2003].codfw.wmnet` - kubetcd2002.codfw.wm... [16:47:21] 10serviceops, 10Deployments, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10thcipriani) >>! In T236104#5916148, @Joe wrote: > We also have the possibility o... [17:36:25] 10serviceops, 10Deployments, 10Performance-Team (Radar), 10Release-Engineering-Team (Deployment services), and 2 others: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 (10Joe) >>! In T236104#5920394, @thcipriani wrote: > Is there a sane workaround we... [18:12:31] 10serviceops, 10Parsoid-PHP, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Create /srv/mediawiki/parsoid-vendor on production MW appservers, a check out of vendor.git's parsoid branch - https://phabricator.wikimedia.org/T245886 (10Jdforrester-WMF) p:05Triage→03High [19:53:59] 10serviceops, 10MediaWiki-General, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team): siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Pchelolo) 05Open→03Declined I have verified that services running internally are cac... [21:15:44] 10serviceops, 10MediaWiki-General, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team): siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Anomie) >>! In T244204#5921402, @Pchelolo wrote: > The only outstanding question left in... [21:20:16] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) mw2291 through mw2300 are in production now, pooled and then set Active in Netbox. [21:20:43] 10serviceops, 10MediaWiki-General, 10Operations, 10Core Platform Team Workboards (Clinic Duty Team): siteinfo api calls should be cached for N minutes on the caching layer - https://phabricator.wikimedia.org/T244204 (10Pchelolo) 05Declined→03Open [21:20:46] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) 05Stalled→03Open a:05Dzahn→03None [22:18:08] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10ssastry) >>! In T245877#5918345, @Dzahn wrote: > @ssastry Yea, though it's not tied to the hostname. It's "the puppet role pa... [22:45:34] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests, 10Patch-For-Review: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Dzahn) @Sbailey @ssastry Sounds good. Now we just need an SSH public key from you. I started a patch to... [22:54:11] 10serviceops, 10Operations, 10Parsoid-PHP, 10SRE-Access-Requests, 10Patch-For-Review: Give all members of the Parsing team production `deployment` access - https://phabricator.wikimedia.org/T245877 (10Dzahn) >>! In T245877#5921857, @ssastry wrote: > Let us start with granting @sbailey access to just the...