[06:51:18] 10serviceops, 10Operations, 10ops-eqiad, 10HHVM: mw1272 crashed: Bad page map in process hhvm - https://phabricator.wikimedia.org/T211668 (10Marostegui) This host crashed today again: ` ------------------------------------------------------------------------------- Record: 40 Date/Time: 02/22/2019 0... [07:49:52] fsero: good morning. I must say I love your envoyproxy commit message at https://gerrit.wikimedia.org/r/#/c/operations/debs/envoyproxy/+/491951/ [07:50:01] each bullet point sounds like a red flag to me ;) [08:55:18] morning hashar [08:55:58] Thanks for adding the jjb config but I don't think it will ever build on our Jenkins [08:56:40] I see is failing because there is no docker installed and cannot run the building image [09:00:02] Envoy is a good piece of software, is just the building process is optimized for ci and based on docker and that doesn't play well within our packaging policies [09:26:43] 10serviceops, 10Multimedia, 10Operations, 10Thumbor, and 2 others: Deploy 3d2png to thumbor servers (stretch) - https://phabricator.wikimedia.org/T216494 (10Gilles) This patch works and I successfully deployed it on thumbor2002 and thumbor1004, where 3d2png now works on Stretch. [09:29:24] fsero: yeah that sounds broken to me :] [09:29:50] fsero: also nutcracker still receive some commits here and ther,e so maybe it is not that much abandoned ;) [09:32:28] maybe I should just add a noop job instead [09:32:41] (ie make CI do nothing but just vote v+2) [09:50:08] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10hashar) @ItSpiderman you are reaching a timeout, notably https://docker-registry.wikimedia.org/v2/_catalog takes more than 30 seconds to return a payload and your docker client reaches a timeout. I don't... [09:50:08] latest fun [09:50:28] akosiaris: fsero: somehow https://docker-registry.wikimedia.org/v2/_catalog takes more than 30 seconds to give a reply :D [09:50:47] seems that prevent at least one user from building a docker image using our registry [09:52:23] hashar: why do they walk the catalog though? [09:52:30] and yes it takes a long time to reply [09:53:07] i am pretty sure it used to be faster [09:53:26] it's not a great endpoint to walk often fwiw [09:53:31] anyway, that user on T216799 seems to reach some timeout [09:53:41] hard to know whether it is on his side or on hour :/ [09:55:41] docker pull works fine on npm-stretch from what I see [09:55:49] ditto :/ [09:57:11] yeah, can't reproduce [09:57:21] I am doing exactly what he just did and it's working [09:57:24] looks like his side [09:57:35] indeed :) [09:57:36] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10hashar) From a discussion with @akosiaris , it seems the registry works fine and both of us can pull just fine. Can you try again by just pulling the image? docker pull docker-registry.wikimedia.or... [09:57:38] thank you! [09:57:59] and especially given that this times out on a specific image and not the /v2/_catalog/ endpoint [09:59:50] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10akosiaris) For what is worth, on a rather flaky internet connection I can just do `docker build --tag quibble .` After git cloning https://github.com/wikimedia/quibble and it works fine. Note that th... [10:03:20] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10akosiaris) @ItSpiderman Could it be something in your docker daemon configuration or your current network, like a proxy? [10:07:48] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10ItSpiderman) I have done the testing, it appears to be an issue related to my system, cURLing https://docker-registry.wikimedia.org/v2/_catalog gives me response (36s), at it is available, while docker p... [10:17:06] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10akosiaris) 05Open→03Invalid Good to know. That you for the update. I 'll mark this as Invalid for now, feel free to reopen though. [11:36:49] 10serviceops, 10Operations, 10Performance-Team, 10Thumbor: Meta Swift container rights incorrect for thumbor user - https://phabricator.wikimedia.org/T216807 (10jijiki) [11:37:22] 10serviceops, 10Operations, 10Performance-Team, 10Thumbor: Meta Swift container rights incorrect for thumbor user - https://phabricator.wikimedia.org/T216807 (10jijiki) [11:37:29] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, 10User-jijiki: Thumbor upgrade to stretch plan - https://phabricator.wikimedia.org/T214597 (10jijiki) [11:46:57] 10serviceops, 10Operations, 10Thumbor: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10jijiki) p:05Triage→03Normal [12:04:06] 10serviceops, 10Operations, 10Thumbor: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10jijiki) [12:04:17] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10jijiki) [12:41:46] 10serviceops: docker-registry.wikimedia.org connection timeout - https://phabricator.wikimedia.org/T216799 (10ItSpiderman) Problem here was that in docker settings DNS server was set to fixed 8.8.8.8 and could not resolve the host. Setting it to automatic solved the issue. **NOTE: Running Docker on Windows!** It... [13:26:34] 10serviceops: docker-registry.wikimedia.org connection timeout on Microsoft Windows (due to DNS server set to "8.8.8.8" in docker settings instead of "automatic") - https://phabricator.wikimedia.org/T216799 (10Aklapper) [18:34:20] 10serviceops, 10Operations, 10MW-1.33-notes (1.33.0-wmf.19; 2019-02-26), 10Patch-For-Review, 10User-Joe: Set up A/B testing mechanism for PHP7, - https://phabricator.wikimedia.org/T216676 (10Krinkle) @Joe I understand the choice between VCL in Varnish and client-side JS favouring the latter. While I'm no... [19:41:15] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10jijiki) [19:43:11] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10jijiki) [19:44:49] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10jijiki) [19:45:14] jijiki: i'm trying to work on the stretch update [19:45:33] jijiki: i see https://gerrit.wikimedia.org/r/#/c/3d2png/deploy/+/491957/ but that seems like it's been done manually [19:53:36] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, and 2 others: Upgrade Thumbor servers to Stretch - https://phabricator.wikimedia.org/T170817 (10jijiki) [19:53:58] jijiki: https://gerrit.wikimedia.org/r/#/c/3d2png/+/492373/ and https://gerrit.wikimedia.org/r/#/c/3d2png/deploy/+/492374/ [19:54:00] you're welcome :) [19:54:47] 10serviceops, 10Operations, 10Performance-Team, 10Thumbor: Meta Swift container rights incorrect for thumbor user - https://phabricator.wikimedia.org/T216807 (10jijiki) 05Open→03Resolved Will reopen if we run into this again, works for now. [19:54:49] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, 10User-jijiki: Thumbor upgrade to stretch plan - https://phabricator.wikimedia.org/T214597 (10jijiki)