[04:30:45] 10serviceops, 10MediaWiki-Maintenance-scripts, 10Operations, 10Patch-For-Review: Stop forcing RUNNER=php for foreachwiki/foreachwikiindblist - https://phabricator.wikimedia.org/T230110 (10Dzahn) 05Open→03Resolved a:03Dzahn [04:30:53] 10serviceops, 10MediaWiki-extensions-Mailgun, 10Operations, 10cloud-services-team, and 5 others: Switch cronjobs on maintenance hosts to PHP7 - https://phabricator.wikimedia.org/T195392 (10Dzahn) [06:21:28] _joe_: i added the envoy user and group in puppet in the "if jessie" section and puppet runs without issues now ununpentium [06:34:12] 10serviceops, 10Operations, 10Traffic, 10Wikidata, and 4 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10Dzahn) >>! In T99531#5414154, @BBlack wrote: > track down various revert patches first before we close it up (revert the DNS repo stuff and w... [06:49:09] <_joe_> mutante: does envoy work there? [06:50:17] _joe_: just found apache is still blocking port 443. going to fix that now [06:50:42] on another host i just unloaded the ssl module.. instead of editing ports.conf [06:50:54] <_joe_> bleargh :P [06:51:12] hmm. i still need it on this host though.. i should use another one to test [06:51:16] looks [06:52:13] maybe the best way forward is to make a stretch ganeti VM and just move RT over [06:56:13] 10serviceops, 10Scap, 10PHP 7.2 support, 10Patch-For-Review, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) @thcipriani should we create a new package/release? [07:30:41] port 443 is now free but envoy has an additional issue. [07:30:51] cant init the config file because "Unable to convert YAML as JSON" [07:31:03] missing a package for it? [07:34:02] no, /etc/envoy/envoy.yaml exists but is empty [07:38:15] <_joe_> mutante: ok ununumpentium? [07:38:20] <_joe_> lemme check what's going on [07:39:04] <_joe_> err ununpentium I meant [07:39:18] _joe_: yep, ununpentium. on stretch /etc/envoy/envoy.yaml is in the output of dpkg -L but on jessie it's not [07:39:31] on jessie it does get created but stays empty [07:40:01] <_joe_> so the weay that gets created is by puppet running build-envoy-config [07:40:09] 10serviceops, 10Operations, 10Traffic, 10docker-pkg: Getting registry metadata from a public client fails on our registry - https://phabricator.wikimedia.org/T220085 (10ema) It seems that CL is returned properly now: ` $ curl -v --http1.1 https://docker-registry.wikimedia.org/v2/python3/manifests/latest 2... [07:40:22] aha [07:40:25] <_joe_> ImportError: No module named 'typing' [07:40:39] <_joe_> omg it's python 3.4 [07:41:57] heh, there is even python 2.7.9 in jessie if you just install python [07:42:09] <_joe_> well this is python3 [07:42:51] yep, 3.4 confirmed in packages.debian.org [07:43:23] <_joe_> so, heh [07:43:37] <_joe_> I need another version of that script I guess for jessie, without the type hints [07:43:51] <_joe_> or, I remove them now, revert once we're free from jessie [07:43:58] <_joe_> but I'm inclined to do the former [07:44:14] that is still easier than converting scb to stretch, right [07:44:30] <_joe_> mutante: scb should just go away and be moved to kubernetes [07:44:38] ack [07:44:49] <_joe_> btw, we need to add a sidecar with envoy to our charts in the future :) [07:45:27] "in Python 3.5 and later, the typing module lives in the stdlib" [07:45:34] one minor version too early [07:46:46] and no jessie for https://packages.debian.org/search?keywords=python-typing [07:48:46] <_joe_> yep [07:51:40] hrmm, ok. then i will let you do what you suggested. i'll focus on moving RT to stretch in a new ganeti VM and then releases1001 to envoy [07:52:09] be back after lunch [07:53:52] let's not migrate [07:54:23] let's not migrate to stretch, if the blocker for moving RT (and other services) is a missing envoy for buster, then let's fix this now? [07:55:34] unless one is replacing an existing server out of a service or well-defined exceptions all new installation should be buster, if there are missing pre reqs, then let's fix them [09:00:35] ok, fair enough. in my mind getting off of jessie was the most important thing [09:01:51] will create a VM with buster to start moving it then (could be with or without envoy at first) [09:03:30] sounds good! [09:11:18] <_joe_> moritzm: I have envoy for buster, btw [09:11:20] <_joe_> but [09:11:26] <_joe_> do we really need to migrate RT?? [09:11:41] <_joe_> we're not using it for anything anymore AIUI [09:12:18] yea, unfortunately. dcops needs it. the alternative plan was to scrape all the HTML and just host that, but cant just delete it [09:12:25] it's needed for some more time to fetch old procurement tasks [09:12:40] until those old servers are completely decommed/out of books [09:12:42] <_joe_> if we could just staticize it... [09:12:47] i wish it would have been imported to private phab tasks [09:13:07] yea, i found a way to do it but it takes me more time i think [09:15:41] <_joe_> well the advantage would be we don't have the db and application to maintain/migrate over and over [09:16:14] <_joe_> but I trust your judgement, I have the jessie fix ready btw [09:16:19] <_joe_> lemme upload the new change [09:18:17] oh, cool! [09:21:11] <_joe_> mutante: I uploaded envoyproxy to buster btw [09:21:39] <_joe_> now I am going to try to write better docs on how to create new packages [09:21:46] nice! so both of them, that was quick [09:22:35] <_joe_> mutante: merging now, so TLS will probably work on ununpentium afterwards [09:22:37] i am already rethinking the rt-static thing.. will follow-up [09:22:46] <_joe_> sure, thanks [09:33:44] <_joe_> mutante: Sep 4 09:31:11 ununpentium envoy[22023]: cannot bind '0.0.0.0:443': Permission denied [09:33:46] <_joe_> ' [09:33:52] <_joe_> uhm I think the problem is [09:34:19] <_joe_> AmbientCapabilities=CAP_NET_BIND_SERVICE doesn't work [09:36:13] yeah, that'll require systemd 230 and jessie has 215 [09:36:20] <_joe_> sigh [09:36:37] <_joe_> in 215 it is Capabilities= [09:37:11] actually, 229, but yeah, not in jessie's systemd [09:37:22] <_joe_> moritzm: but I think it just changed names [09:38:26] <_joe_> [/lib/systemd/system/envoyproxy.service:15] Failed to parse capabilities, ignoring: CAP_NET_BIND_SERVICE [09:38:28] <_joe_> sigh [09:38:40] <_joe_> that wasn't included still, indeed. [09:38:53] <_joe_> so on jessies, we need to use a different port. [09:38:55] yeah, indeed.from the 230 changelog: [09:39:11] The Capabilities= unit file setting has been removed (it is ignored for backwards compatibility). AmbientCapabilities= and CapabilityBoundingSet= should be used instead. [09:39:24] at least it's ignored for backwards compat !!1! [09:39:25] <_joe_> yeah, but still [09:40:48] <_joe_> using Capabilities= it should work [09:41:36] <_joe_> and man 7 capabilities on the server includes CAP_NET_BIND_SERVICE [09:43:13] <_joe_> Take a capability string describing the effective, permitted and inherited capability sets as documented in [09:43:15] <_joe_> cap_from_text(3) [09:43:23] <_joe_> ohhh sigh it's not as simple as I thought [09:43:34] <_joe_> mutante: can you please use a non-privileged port? :P [09:51:24] <_joe_> so FTR, I tried setting [09:51:52] <_joe_> Capabilities="cap_net_bind_service=+ep" but still doesn't get parsed [09:52:03] <_joe_> moritzm: any idea what I might have done wrong? [09:53:09] <_joe_> ok found it [09:53:27] <_joe_> I did already correct my error (adding the =) but I had to remove the double quotes [10:06:45] <_joe_> mutante: it's definitely a pain, we should move to an unprivileged port. [10:14:21] _joe_: ok! i think earlier i saw ema's changes that allow changing it? [10:14:35] <_joe_> read above :P [10:14:55] <_joe_> and yes, it's now not bound to port 443 [10:15:11] yea, but it wasn't even possible unless that is already merged? [10:15:20] <_joe_> I think it is [10:16:30] looks at https://gerrit.wikimedia.org/r/c/operations/puppet/+/534184 [10:17:05] <_joe_> mutante: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534401 does some cleanup on jessie [10:17:48] alright [19:47:00] 10serviceops, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move old transparency report pages to historical URLs and setup redirect - https://phabricator.wikimedia.org/T230638 (10APalmer_WMF) Hi all, just wanted to see if there was any further info or clarification needed from Legal. We really appreciate... [19:52:22] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [19:58:14] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) p:05Triage→03Unbreak! I guess I will poke wmde/serviceops tomorrow morning to f... [20:06:57] anyone have any guidance on troubleshooting k8s deployment failures (staging)? The original gerrit was https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/533922 which resulted in https://phabricator.wikimedia.org/P9041 [20:07:59] that made me think a pod couldn't have memory set to less than 100Mi (and in fact the gerrit dropped it from 100Mi to 50Mi), so I bumped it back up, but no joy [20:09:49] I think my track record here is like 1 in 5 (1 deploy in 5 succeeds) [20:38:11] 10serviceops, 10Scap, 10PHP 7.2 support, 10Patch-For-Review, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10thcipriani) >>! In T224857#5463992, @Joe wrote: > @thcipriani should we create a new package/release? yes! The scap repo has tags f... [21:12:25] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Ladsgroup) [21:26:47] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Ladsgroup) The link to logs: https://logstash.wikimedia.org/goto/3146ff65c2320362e12501bb1f... [23:34:24] 10serviceops, 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell) [23:35:38] 10serviceops, 10Phabricator, 10Release-Engineering-Team-TODO (201909), 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell) p:05Triage→03High [23:55:20] 10serviceops, 10Phabricator, 10Release-Engineering-Team-TODO, 10User-MModell: Mukunda to set up a meeting with service ops to discuss operational best practices for phabricator - https://phabricator.wikimedia.org/T232058 (10mmodell)