[04:10:55] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10docker-pkg: service-pipeline-test jenkins job fails due to jessie-backports removal - https://phabricator.wikimedia.org/T219580 (10greg) Does the wikimedia-jessie base image need to be updated? [10:04:07] <_joe_> moritzm, jijiki:: I would love to get your feedback on ways out of https://phabricator.wikimedia.org/T219279 [10:04:52] <_joe_> it's a really big mess, the TL;DR: utf-8 broken in php until 7.2; now fixed, means we had users and pages that were not properly normalized, so that now they'd be unreachable [10:05:48] <_joe_> it's like: for mediawiki, Joe and joe are the same user, but until php 7.2 ßoe and Ᏼoe weren't [10:07:08] oh dear [10:14:29] <_joe_> so yeah, fun! [10:25:11] this sounds like fun, I'll have a look at the task later on [10:26:48] function mediawiki::to_upper() #HACK because PHP is wrong [10:27:29] in related news, akosiaris will be delighted to hear that with Debian buster we'll not only see collation changes in mediawiki, but also in postgres: https://lists.debian.org/debian-glibc/2019/03/msg00030.html [10:34:24] hahahaha [10:35:08] I should not be laughing [13:51:31] about this task [13:51:34] https://phabricator.wikimedia.org/T219580 [13:51:56] should we consider to start using ste [13:52:02] stretch for this job ? [13:57:24] <_joe_> I guess so [13:57:30] <_joe_> and greg's comment is correct [13:57:37] <_joe_> we need to change the base image I guess [13:58:04] <_joe_> and to rebuild anything based on it [13:59:03] <_joe_> jbond42: ^^ we might be in for more surprises [13:59:53] <_joe_> as in: packages from backports used in images [14:05:49] is the faliure just that missing upstream os is it allso missing a package? [14:06:25] sorry missing upstream repo, or do the images need a package from backports which is missing [14:15:52] moritzm: yeah I 've that already... it's a mess [14:15:57] seen* [14:16:30] the "bug" has been known for years, it's just recently that more obvious corruptions started being known [14:17:01] there was beatiful case where they would cause a duplicate key insertion because of collation mismatches on the index [14:23:16] <_joe_> jbond42: I don't know (yet) [14:23:26] <_joe_> I'm going to work on that ticket in a few [14:25:49] ok i have added my self as a subscriber so will keep an eye on things, ar you able to point me in the direction of some documentation on how images are managed? [14:38:19] * _joe_ coughs [14:38:28] <_joe_> jbond42: it's a work in progress [14:38:35] <_joe_> but yes, I need to write a wiki page too [14:38:45] :D ok [14:43:50] <_joe_> oh my gerrit is sloooow [14:45:28] <_joe_> jbond42: https://gerrit.wikimedia.org/r/500054 is the solution of the immediate bug, but I'll need to inspect the child images [14:46:49] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10docker-pkg, 10Patch-For-Review: Remve backports from wikimedia-jessie - https://phabricator.wikimedia.org/T219580 (10Joe) [14:47:18] _joe_: are you sure i would have expected you to need to remove jessie-backports from line 34 [14:47:28] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10docker-pkg, 10Patch-For-Review: Remve backports from wikimedia-jessie - https://phabricator.wikimedia.org/T219580 (10Joe) a:03Joe [14:47:36] <_joe_> oh you're right [14:47:51] <_joe_> meh, friday afternoon [14:47:58] :) [14:48:02] <_joe_> backports should actually stay there [14:48:47] yes backports now has the packages from jessie-backports that where in use in production [14:49:16] <_joe_> yeah I just got it completely backwards when actually removing things from the file [14:49:17] however there could still be issues there as well i think [14:49:32] <_joe_> I did read the task with your solution [14:49:37] ok [14:49:41] <_joe_> you mean missing packages? [14:50:01] there could be dependencies pulled in from jessie-backports which did not get moved accross [14:50:23] <_joe_> yeah I need to inspect the child images, which are in two main repositories: integrations/config AND operations/docker-images/production-images [14:50:38] <_joe_> on and ofc toollabs-images or however that's called [15:01:31] <_joe_> jbond42: can I ask you a quick +1 on that patch? I want to go on and fix CI [15:02:15] yes one sec [15:02:24] <_joe_> sure [15:04:10] done [16:19:20] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10docker-pkg, 10Patch-For-Review: Remove backports from wikimedia-jessie - https://phabricator.wikimedia.org/T219580 (10Aklapper) [16:43:47] 10serviceops, 10Release Pipeline, 10Release-Engineering-Team, 10docker-pkg, 10Patch-For-Review: Remove backports from wikimedia-jessie - https://phabricator.wikimedia.org/T219580 (10Joe) This is harder than expected, as bootstrap-vz automatically adds `{release}-updates` to `/etc/apt/sources.list`, see... [16:59:23] 10serviceops, 10Release Pipeline, 10docker-pkg, 10Patch-For-Review: Remove backports from wikimedia-jessie - https://phabricator.wikimedia.org/T219580 (10greg) [17:09:32] 10serviceops, 10Operations: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10jijiki) [17:12:53] 10serviceops, 10Operations, 10Services (watching): Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10Pchelolo) [17:23:41] 10serviceops, 10Operations, 10Services (watching): Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10jijiki) [17:29:47] 10serviceops, 10Operations, 10Services (watching), 10User-jijiki: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10jijiki) [17:31:26] 10serviceops, 10Operations, 10Services (watching), 10User-jijiki: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10Pchelolo) If I understand correctly, in order to switch a particular job execution to PHP7 all we need to do is to add `Cookie: PHP_ENGINE=php7` header to the requ... [19:54:57] 10serviceops, 10Operations, 10Core Platform Team Backlog (Watching / External), 10Services (watching), 10User-jijiki: Use PHP7 to run all async jobs - https://phabricator.wikimedia.org/T219148 (10mobrovac) [22:34:36] 10serviceops: Rebuild docker-registry.wikimedia.org/wikimedia-jessie to drop jessie-update/jessie-backports - https://phabricator.wikimedia.org/T219683 (10hashar)