[00:06:38] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar): Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10aaron) The current callers don't assume the level of durability as with mysql, just that the da... [00:13:11] rsyncing all the /srv/org/wikimedia/docs from contint1001 to doc1001 now, still using !log in operations channel only, i assume we want to keep it that way [06:43:09] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar): Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10akosiaris) >>! In T212129#4833043, @aaron wrote: > The current callers don't assume the level o... [07:18:05] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar): Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10Joe) Looking at live data, we have at least one shard that's doing evictions (150k of them) and... [08:27:20] 10serviceops, 10Operations, 10TechCom, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) Using accept-language is not an option, at least not the accept-language from the browser. The relevant list of languages comes from user prefere... [08:32:49] 10serviceops, 10Operations, 10TechCom, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) I agree with Joe that it would be better to have the service be internal, and be called from MW. It doesn't //have// to be that way, but it's pre... [08:55:29] 10serviceops, 10Operations, 10TechCom, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Joe) >>! In T212189#4833482, @daniel wrote: > I agree with Joe that it would be better to have the service be internal, and be called from MW. It doesn't... [09:10:22] 10serviceops, 10Operations, 10TechCom, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10daniel) > Well, I consider calling the MW api from a service called by MediaWiki an antipattern that we should absolutely avoid. Oh, I got that wrong -... [09:56:22] 10serviceops, 10Operations, 10vm-requests, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) With [[ https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480573/ | Gerrit #480573 ]] doc1001.eqiad.wmn... [10:42:17] i need some help creating the VM, im clearly doing something wrong [10:42:22] https://www.irccloud.com/pastebin/BwqHzUbH/ [10:43:13] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10hashar) the `wmnet.` top level domain is for production / internal services. WMCS is considered an alien and is not allowed to reach it. The s... [11:34:41] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) I guess it depends on the use cases. As SREs I don't think we would like to expose a public service that is not going to receive a... [12:08:07] <_joe_> fsero: why not using makevm? [12:08:33] <_joe_> also, the error is in your paste [12:08:34] im using makevm its just it was missing the DNS entries and hence fail [12:08:49] <_joe_> yeah I was about to say that :P [12:08:50] makevm outputs the gnt command := [12:09:00] <_joe_> ok, never used it :D [12:09:00] im fixing the DNS part already _joe_ [12:09:12] thanks [12:09:32] it's just editing a large file with the potential of destroy several things [12:09:47] make me feel *things* [12:10:36] <_joe_> well code review and CI are here to help you [12:11:11] <_joe_> don't worry, if you add a wrong dns entry worse that can happen is you win a t-shirt [12:34:54] merging https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480481/ [14:26:06] ok [14:26:11] we need to post our goals by end of today [14:26:16] where are we on deployment pipeline finalization? [14:43:36] akosiaris: ^ [14:44:58] fsero: I 'll left a comment for "and enable some cluster features like pod autoscaling" in https://etherpad.wikimedia.org/p/SRE-goals-FQ3-FY1819 [14:45:28] mark: are these final or still draft ? [14:45:41] I have one question for petr yet [14:51:39] well [14:51:43] they should become final after today [14:52:31] the only thing in question is whether we should move one of changeprop/cpjobqueue into next quarter. [14:53:24] and the "and enable some cluster features like pod autoscaling" which I think is too specific for our wording. Sure let's set the groundwork for it, but no need to be in the goal [15:04:20] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10hashar) >>! In T212251#4833906, @akosiaris wrote: > I guess it depends on the use cases. > > As SREs I don't think we would like to expose a... [15:07:59] akosiaris: lgtm if we keep an eye on working on that [15:08:36] of course. it's not like we only do what's on the goals. It's a guide and a roadmap but there is always other work being done [15:08:57] and working on the hpa is something we 've being eyeing since the start [15:10:42] yeah let's leave that off [15:11:02] as for changeprop/cpjobqueue... [15:11:15] I can reread the thread [15:12:36] but at least we can post the goal [15:12:48] we can tweak those services still, wouldn't affect the rest right [15:13:01] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) >>! In T212251#4834325, @hashar wrote: >>>! In T212251#4833906, @akosiaris wrote: >> I guess it depends on the use cases. >> >> As... [15:13:49] yes [15:14:09] we can anyway migrate any other service if we have the time [15:14:15] eventstreams being a pretty good candidate [15:16:06] so let's keep it off [15:16:11] we'll do what we reasonably can [15:16:18] but I fully agree that documentation/training is more important [15:19:20] so please update the draft language in the etherpad [15:19:24] then i'll post it [15:26:33] Done [15:37:16] our goal covers multiple outcomes in TEC3 really... [15:37:26] there's a separate outcome for developer portal stuff [15:40:35] and a separate one for improving the k8s cluster itself [15:40:40] so really it should be 3 separate goals :P [15:40:54] i guess i'll do that [15:41:01] and we'll handle it as one big goal internally [15:44:22] visual editor is not really working well on mediawiki.org [15:44:55] it often doesn't fully load [15:54:52] akosiaris: ok our goal is all over https://www.mediawiki.org/w/index.php?title=Wikimedia_Technology/Annual_Plans/FY2019/TEC3:_Deployment_Pipeline/Goals [15:54:57] check if I got that correct :P [15:57:39] ah I had forgotten that format [15:59:51] LGTM [16:01:04] <_joe_> akosiaris: your ability to forget trauma is remarkable [16:01:34] it's part of the healing process [16:19:26] Hi, wondering if someone could review https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480760/ please? :) (i've tested it locally and works) [16:19:57] Also i think we can drop sysvinit now that shinken in the cloud is on jessie [16:23:22] akosiaris: _joe_ so if I merge the DNS change and go over ns0 and run authdns-update [16:23:29] Everything should be good right? [16:23:46] Iyes [16:23:53] fsero: yes [16:24:41] ok I'll do that after coffee [16:38:40] _joe_: if that was trauma [16:38:47] wait until I _really_ involve you in the process :P [17:15:06] 10serviceops, 10Operations, 10vm-requests, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) The VM is working and the basic service is there ( rsyncd ). I will complete the service implementation via... [17:54:32] working on the new doc.wikimedia.org with hashar, it's all in -releng for now ^ [18:36:51] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team (Kanban): Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10dduvall) >>! In T212251#4834349, @akosiaris wrote: >>>! In T212251#4834325, @hashar wrote: >>>>! In T212251#4833906, @akosiaris wrote: >>> As S... [18:44:07] 10serviceops, 10Operations, 10vm-requests, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) shell access for existing groups contint-admins and contint-users has been granted (same access people had be... [18:45:16] 10serviceops, 10Continuous-Integration-Infrastructure, 10Developer-Wishlist (2017), 10Patch-For-Review, and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Dzahn) [19:46:31] 10serviceops, 10Operations, 10vm-requests, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) 05Open→03Resolved Yes, the VM has been created, basic role has been created, users added, httpd installed... [19:46:34] 10serviceops, 10Continuous-Integration-Infrastructure, 10Developer-Wishlist (2017), 10Patch-For-Review, and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Dzahn) [19:47:04] 10serviceops, 10Operations, 10vm-requests, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) [19:48:53] 10serviceops, 10Operations, 10vm-requests, 10Release-Engineering-Team (Kanban): eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) Thank you for the quick spinning of the instance as well as all the preliminary puppet work. Much appreciated :) [19:55:20] what do i have to do to tell the puppet compiler about new hosts? f.e. i create a new host and want to test things on it right away and compiler does not know the host yet. How do i sync the facts? [20:06:40] 10serviceops, 10Operations, 10TechCom, 10Wikidata, and 5 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Milimetric) In https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/SSR_Service we see that "There is a server-side and the client-side variant of the code,... [20:07:53] 10serviceops, 10Core Platform Team, 10MediaWiki-Cache, 10Operations, 10Performance-Team (Radar): Use a multi-dc aware store for ObjectCache's MainStash if needed. - https://phabricator.wikimedia.org/T212129 (10Eevans) >>! In T212129#4834329, @mark wrote: > I am getting the impression here that some thing... [20:27:35] paladox: should the "SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1" not be outside the "if fpm is enabled" block as unrelated? [20:36:30] paladox: just noticed the SetHandler line is different in my case vs your existing phab case. /run/php/fpm-www.sock vs /run/php/php-7.2-fpm.sock [20:38:20] mutante: that’s needed to get login to work when using php-fpm [20:38:24] At least I found that [20:39:20] oh really it's related? interesting [20:40:08] Yeh I found some problem that I think was logins not working [22:05:48] 10serviceops, 10Continuous-Integration-Infrastructure, 10Developer-Wishlist (2017), 10Patch-For-Review, and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10hashar) The scap config for integration/docroot is: * https://gerrit.wikimedia.org/r/#/c/integ... [22:20:52] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Assess Thumbor upgrade options - https://phabricator.wikimedia.org/T209886 (10jijiki) @kaldari We have deployed librsvg 2.40.20-3 on deployment-imagescaler03 under debian stretch, after some testing it does't look like SVG rendering... [23:35:12] ok, figured out how to sync the compiler facts and just did. it finds my new host. solved [23:53:30] mutante: FYI https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs#FAQ (it redirects also from https://wikitech.wikimedia.org/w/index.php?title=Puppet_Compiler&redirect=no ) [23:54:59] thanks, that's what i used (after some confusion about puppet3-diffs and the old compiler hosts) [23:58:44] volans: enjoy vacation, good travels