[00:11:06] <wikibugs>	 10serviceops, 10Operations, 10User-jijiki: mw2225 keeps sending cronspam for hhvm-needs-restart - https://phabricator.wikimedia.org/T236799 (10Dzahn) Mysterious. I could not find "hhvm-needs-restart" anywhere. Not in any crontab, not in systemd timers.. not anywhere else in /etc....
[00:13:36] <wikibugs>	 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Dzahn) >>! In T236275#5642688, @Arlolra wrote: > `restart_php` failed, yet scap reported all good `(ok: 2; fail: 0; left: 0)`  The  reason why it fa...
[07:51:01] <wikibugs>	 10serviceops, 10Parsing-Team, 10Parsoid, 10PHP 7.2 support: Parsoid-php doesn't get updated after a code deploy - https://phabricator.wikimedia.org/T236275 (10Joe) 05Open→03Resolved It is expected you see that failure as one of the two servers you're deploying to in beta, `deployment-parsoid09.deployme...
[09:47:34] <wikibugs>	 10serviceops, 10Operations, 10Patch-For-Review, 10Puppet, 10User-jbond: Rolling restart of etcd to pick up the renewed CA public certificate. - https://phabricator.wikimedia.org/T237362 (10Joe) Correction:  # We will need to restart etcd in eqiad as the CA is used in etcd::v3 for peer-to-peer communicati...
[10:01:11] <jynus>	 this confused me for a bit: https://phabricator.wikimedia.org/P9547
[10:21:50] <_joe_>	 scope goes before the verb
[10:22:06] <_joe_>	 yes it's a bit confusing, I blame python's ArgumentParser for that
[10:22:26] <_joe_>	 dbctl -s eqiad <command>
[10:23:44] <_joe_>	 jynus: if you'd prefer it not to be tied to happen before the <command> switch, can you open a quick task?
[10:24:23] <jynus>	 I percisely didn't not open a task because I do not have a better suggestion
[10:24:43] <jynus>	 I think it is my expectations based on how git works
[10:25:33] <_joe_>	 so one suggestion could be to allow -s to go anywhere after the first positional argument
[10:25:51] <_joe_>	 or in any position really
[10:26:01] <_joe_>	 implementing that in python is just boring, but can be done
[10:26:01] <jynus>	 please don't spend time on this
[10:26:21] <jynus>	 I just wanted to speak up and share it, but no actionable needed
[15:17:59] <subbu>	 _joe_, on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/548923 .. i imagine you meant appserver right .. not specific endpoints.
[15:18:23] <subbu>	 append a "?" to the end of that line. :)
[15:23:52] <_joe_>	 heh, debatable :)
[15:24:16] <_joe_>	 I think we should, on the same server, reserve different amount of resources to different endpoints indeed
[15:24:42] <_joe_>	 but it can be applied, in a more shallow way, just to appservers by function group
[15:36:56] <subbu>	 ok :)
[16:09:25] <ottomata>	 akosiaris:  o/
[16:09:36] <ottomata>	 couple of qs if you are around:
[16:09:42] <ottomata>	 1. namespaces plz! :D
[16:09:56] <ottomata>	 2. do you think I should create a discovery URL for this schema.svc?
[16:10:00] <ottomata>	 it isn't strictly necessary
[16:10:19] <ottomata>	 from external cache will route to proper dc .svc
[16:10:30] <ottomata>	 and internal can be configured with the .svc URLs
[16:10:36] <ottomata>	 but, perhaps discovery is just a better practice?
[16:16:10] <_joe_>	 ottomata: discovery is handy if you need to depool one dc
[16:16:22] <_joe_>	 as in - for your application
[16:16:30] <_joe_>	 but I don't think you do, strictly speaking
[16:16:38] <ottomata>	 yeah, its just static files
[16:16:43] <ottomata>	 so not really a big need
[16:16:44] <_joe_>	 ottomata: it's hosted on k8s??
[16:16:47] <ottomata>	 no
[16:16:48] <ottomata>	 ganeti
[16:16:49] <_joe_>	 please tell me no 
[16:16:52] <_joe_>	 ok :)
[16:17:09] <_joe_>	 ok then no reason really
[16:17:18] <ottomata>	 any reason for ats?
[16:17:19] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/549177/1/hieradata/common/profile/trafficserver/backend.yaml
[16:17:21] <_joe_>	 it's a mostly independent system with ridicule resource uage
[16:17:22] <ottomata>	 what would I put here?
[16:17:38] <_joe_>	 ottomata: does schema has tls termination?
[16:17:50] <ottomata>	 not itself no
[16:18:00] <ottomata>	 its just an nginx http server, no special stuff
[16:18:09] <ottomata>	 it could i suppose but i'd hope that would be taken care of by frontend
[16:18:28] <_joe_>	 we're moving towards TLS everywhere
[16:18:51] <ottomata>	 backend apps should terminate?
[16:19:25] <ottomata>	 at the internal .svc url?
[16:21:08] <ottomata>	 _joe_ ^?
[16:21:42] <_joe_>	 ottomata: not sure I understand
[16:21:47] <ottomata>	 me neither
[16:21:55] <ottomata>	 "we're moving towards TLS everywhere"...yes
[16:21:58] <_joe_>	 ottomata: using TLS for all communications between internal applications
[16:22:10] <_joe_>	 so that they can also live across datacenters
[16:22:18] <_joe_>	 and talk to each other without interference
[16:22:20] <ottomata>	 should the nginx http server on  the backend do https termination?
[16:22:31] <ottomata>	 i thought there was frontend  routing that did termination
[16:22:33] <_joe_>	 schema is kind of a special case, being only static files
[16:22:43] <_joe_>	 ottomata: for the external clients, yes
[16:22:52] <_joe_>	 but now ATS also uses TLS to connect to the backend
[16:23:19] <ottomata>	 interesting.  
[16:23:28] <ottomata>	 is that abstracted away for k8s services?
[16:23:30] <_joe_>	 if service A needs to speak to service B, we want that communication to be encrypted
[16:23:35] <_joe_>	 ottomata: in the process of
[16:23:38] <ottomata>	 cool
[16:23:40] <_joe_>	 with a TLS sidecar
[16:23:42] <ottomata>	 nice
[16:23:58] <_joe_>	 see the recent changes to scaffolding, my goal this Q is to add it to all existing services on k8s
[16:24:22] <ottomata>	 ok so should I figure out https for my nginx server then?
[16:24:41] <ottomata>	 how does that work with lvs?  i guess it doesn't matter?
[16:25:02] <_joe_>	 so for this, I will tell you to chat with ema
[16:25:11] <_joe_>	 and yes, you'll need a second load-balanced port
[16:28:41] <ottomata>	 ok
[18:43:59] <wikibugs>	 10serviceops, 10Operations, 10observability, 10Performance-Team (Radar): Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal - https://phabricator.wikimedia.org/T234283 (10Krinkle) 05Open→03Resolved
[18:44:02] <wikibugs>	 10serviceops, 10Operations: SRE FY19-20 Q1 goal: complete the transition to PHP7 - https://phabricator.wikimedia.org/T219127 (10Krinkle)
[18:44:09] <wikibugs>	 10serviceops, 10Operations, 10observability, 10Performance-Team (Radar): Messages in Logstash from php-fatal-error.php are missing from type:mediawiki/channel:fatal - https://phabricator.wikimedia.org/T234283 (10Krinkle) Thanks, confirmed here as well.