[06:40:39] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) [07:03:19] can we limit connections to envoy to CACHE servers? https://gerrit.wikimedia.org/r/c/operations/puppet/+/534421 [07:09:50] <_joe_> mutante: no [07:10:36] ok [07:11:39] also found we still had the port hardcoded in the ferm rule. fixed by ema https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534442 [07:12:01] <_joe_> didn't ema fix it? [07:12:21] <_joe_> right :P [07:12:25] yea, after i commented on that first change [07:12:29] <_joe_> yeah it was his fault anyways [07:12:35] <_joe_> :D [07:14:55] i am using port 1443 now. it seemed to work but after the next restart i run into failed " (Result: start-limit)" [07:15:33] sounds like i just need to be slower [07:15:50] looking if we use any StartLimit options [07:16:02] <_joe_> not sure that's the case [07:16:16] <_joe_> anyways, ttyl [07:16:28] ttyl, i'll try to figure it out [07:17:31] <_joe_> [2019-09-05 07:17:18.624][30586][critical][assert] [external/envoy/source/server/hot_restart_impl.cc:45] panic: cannot open shared memory region /envoy_shared_memory_0 check user permissions. Error: File exists [07:17:42] <_joe_> found using sudo -u envoy /usr/bin/envoy -c /etc/envoy/envoy.yaml [07:18:04] <_joe_> $ ls -la /dev/shm/envoy_shared_memory_0 [07:18:06] <_joe_> -rw------- 1 root root 104 Sep 5 07:08 /dev/shm/envoy_shared_memory_0 [07:18:13] <_joe_> just remove that [07:18:24] ah, thanks [07:18:44] <_joe_> that's the shm space it uses for the hot-restart [07:18:49] <_joe_> which we shall activate soon [07:18:56] aha [07:18:59] * _joe_ brb [07:20:04] next: cannot bind '0.0.0.0:443': (i already manually deleted the listener for 443 though since puppet does not remove that) ..digging [07:21:06] yea, that is still in envoy.yaml.. gotta get it regenerated [07:24:41] removed envoy.yaml. ran puppet. stayed empty. ran build-envoy-config manually. -> TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' [07:27:05] ok, /usr/local/sbin/build-envoy-config -c /etc/envoy works [07:28:50] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Tarrow) I think this is probably the same as T229313. We suspected it might be related to T... [07:29:57] Active: active (running) since Thu 2019-09-05 07:28:00 UTC; 1min 50s ago [07:30:00] yay :) [07:34:52] _joe_: Don't suppose you are around? [07:35:27] those timeouts from termbox on cofdw seems to have suddenly got worse with wmf21 and are now blocking the train [07:35:55] If you happened to have a moment could you take a look at https://phabricator.wikimedia.org/T232035 [07:40:07] <_joe_> tarrow: I disagree with your assessment. This is a release problem. Those past slowdowns have gone away , too :) [07:40:10] <_joe_> sorry, brb [07:40:37] cool! That's certainly useful [07:41:01] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Joe) >>! In T232035#5467309, @Tarrow wrote: > I think this is probably the same as T229313.... [07:58:38] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Some debugging thingies, reaching out to https://www.wikidata.org/w/index.php?titl... [08:12:48] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Release, and 3 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Tarrow) I tried the same from codfw's deployment host and saw the same. No obvious differen... [08:26:06] 10serviceops, 10Scap, 10PHP 7.2 support, 10Patch-For-Review, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) I did some tests, and we still have one problem with `scap pull`: - It is run as a common user (e.g. `foo`) - It runs commands... [08:26:43] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Patch-For-Review, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Promoted wikidatawiki again and I ran the service checker again on deploy1... [08:52:17] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Patch-For-Review, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Looking at `icinga_contact.raw: "irc"`, `Termbox` over 15 days https://log... [08:55:37] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Patch-For-Review, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10Joe) So the real issue was: - termbox **correctly** uses the `api-ro.discovery.wmn... [08:58:18] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Patch-For-Review, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) Rolled back to 1.34.0-wmf.20. The latency is similar ~ 750ms https://graf... [09:01:35] _joe_: so to summarize: envoy is now running on ununpentium as an example for jessie. then i disabled puppet and removed the ferm hole and requested a new VM to replace it (buster and private IP) [09:02:07] <_joe_> nice! [09:02:19] <_joe_> btw it's ok if people just connect directly to it I think [09:02:24] <_joe_> it's not that big of a deal [09:03:33] ok, good [09:05:27] yeah, only a handful of people in SRE can access it anyway [09:05:46] 10serviceops, 10Wikidata, 10Wikidata-Termbox, 10Patch-For-Review, and 4 others: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500 - https://phabricator.wikimedia.org/T232035 (10hashar) 05Open→03Resolved a:03hashar Fixed with the help of @tarrow and @Joe... [09:10:25] found the perfect name for it, chemical element we never used before and is the real name until the "placeholder name" is confirmed. Moscovium "In 1979 IUPAC recommended that the placeholder systematic element name ununpentium (with the corresponding symbol of Uup)[26] be used until the discovery of the element is confirmed and a permanent name is decided. " heh [21:13:30] 10serviceops, 10Performance-Team, 10Scap, 10Continuous-Integration-Config, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) Tagging scap as this implies changes to scap's code.