[12:53:17] valhallasw`cloud: yeah, it’s weird that nothing is there but helpful for today and tomorrow :) [13:14:25] 6Labs, 6operations, 10wikitech.wikimedia.org: Figure out what to do about maintenance scripts on silver/wikitech - https://phabricator.wikimedia.org/T107547#1552732 (10Krenair) >>! In T107547#1551799, @Dzahn wrote: > Isn't the solution here to add the appropriate firewall holes to resolve T98682 instead of l... [14:52:23] Ops: I’m going to reboot bastion-restricted-01 in 10 minutes. That’s the bastion that all ops use to access labs instances. [14:56:40] o.O [15:22:00] YuviPanda|afk: tomorrow’s reboot includes tools-redis-01. Can you switch things over to -02 so that that doesn’t cause interruption? [15:30:30] 10Tool-Labs-tools-Erwin's-tools: mysql library deprecated - https://phabricator.wikimedia.org/T109591#1553080 (10Akoopal) 3NEW [16:00:39] Hi, good day, the xtools is inactive :https://tools.wmflabs.org/xtools-ec/?user=Leon+saudanha&project=en.wikipedia.org ... can solve this? or is some maintenance being done on it? [16:00:57] andrewbogott: hmm the switchover isn't seamless since things will have to break and reconnect anyway... [16:01:22] YuviPanda: does that mean I should just reboot redis-01 and not worry about it? [16:01:23] andrewbogott: wondering if we should be better off just bringing it back up first [16:01:55] andrewbogott: yeah. From a user perspective those might be similarly seen. Both a switchover and a reboot [16:02:00] ok [16:02:19] Do you want to send an additional announcement about the symptoms, or do you think existing emails are enough? [16:03:05] andrewbogott: I'll send an additional email soon. This is tomorrow right? [16:03:13] YuviPanda: yes, tomorrow [16:05:46] Sorry again, but the xtools is inactive :https://tools.wmflabs.org/xtools-ec/?user=Leon+saudanha&project=en.wikipedia.org ... can solve this? or is some maintenance being done on it? [16:06:24] Le0n: hello! Please contact the xtools maintainers - cyberpower and ask them about it? [16:06:38] Le0n: there is also an xtools mailing list I think [16:10:55] (03PS4) 10Sitic: Add support for subdivided watchlists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/231789 (https://phabricator.wikimedia.org/T109188) [16:22:44] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1553498 (10Dzahn) The title says "from tin, terbium, etc". Could you specify the "etc" part of that? [16:28:53] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1553507 (10Krenair) So we may want it to be accessible from certain other DB hosts (T89548), maybe snapshot hosts too (for dumps)? I think this would techn... [16:34:33] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database, 5Patch-For-Review: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1553518 (10Dzahn) There are 2 steps here: a) allow connection per firewall rules b) grant permission in mysql itself a) needs an... [16:40:37] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database, 5Patch-For-Review: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1553545 (10jcrespo) b) Depends on knowing a), as users in MySQL have an srange, too. I suppose you want the standard production users. [16:45:49] 6Labs, 6operations, 10wikitech.wikimedia.org: Figure out what to do about maintenance scripts on silver/wikitech - https://phabricator.wikimedia.org/T107547#1553629 (10Dzahn) I think with "add firewalling -> notice that it breaks things" it's natural to adjust the firewall. [16:46:52] marxarelli: random mediawiki-vagrant question of the day. I want to add detection in Vagrantfile of vagrant being run on a labs host so I can change the 'environment' puppet fact to 'labs' instead of the default 'vagrant'. Any great ideas on how to do that? [16:47:35] We have a puppet role that sets things up so I guess it could make a signal file of some sort [16:47:37] hmm [16:47:50] yeah, having puppet signal to the install seems cleaner [16:48:30] oh, maybe I can do it nicely in the /usr/local/bin/mwvagrant exec wrapper actually [16:48:48] via an ENV setting. then I could test it other places too [16:49:09] something like MWV_ENVIRONMENT=foo [16:49:19] is there some sort of universal I_AM_LABS environment variable? [16:49:22] and then a check for that in the Vagrantfile with a default [16:50:09] yeah, that would work. check for that environment variable and override the `environment` fact in Vagrantfile [16:50:35] 'environment' => ENV['MWV_ENVIRONMENT'] || 'vagrant' [16:50:47] *nod* [16:51:21] There is sort of a magic "I'm in labs" env var -- INSTANCENAME [16:51:32] but I think the other way is nicer [16:51:42] a bit more explicit [16:52:08] so you have to use the mwvagrant wrapper when using it in labs? [16:52:40] yes, but there is an alias in /etc/profile.d for it to `vagrant` [16:53:03] that really mixed me up the other day [16:53:13] ah, ok. as long as there won't be inconsistent entry points, i think that's a decent solution [16:53:29] the wrapper does all the right sudo stuff for shared control of the /srv/mediawiki-vagrant dir and the vm [16:53:33] because `ssh foo.eqiad.wmflabs VAGRANT_CWD=/srv/mediawiki-vagrant/ vagrant ssh -- ls` runs the wrong vagrant :P [16:53:43] ebernhardson: oh yuck [16:53:57] because no profile.d for that I bet [16:54:02] bd808: yes, also forwarding ssh multiple levels deep is nasty [16:54:18] but what i realized what the alias doesn't come in, because yea no profile.d [16:54:41] the second ssh isn't really ssh either ;) [16:54:52] vagrant jsut says it is [16:55:17] it's really an lxc-attach call [16:55:20] what a cheater [16:56:22] so the right call there is `ssh foo.eqiad.wmflabs /usr/local/bin/mwvagrant ssh -- ls` [17:30:59] YuviPanda: about xtools being down that Le0n reported... I guess the automatic restart service thing isn't working for that tool? [17:31:17] any reservations about me re-enabling the webwatcher script? if you recall [17:39:22] MusikAnimal: yes, about two gazillion reservations [17:39:28] as is explicitly mentioned in the crontab [17:40:35] dah, sorry was not aware of the phab task [17:40:44] that is interesting [17:43:20] valhallasw`cloud: I wonder how that happens? the script just calls `webservice restart` [17:44:58] YuviPanda: around? [17:46:38] MusikAnimal: yes, but it runs it every few minutes, so if an edge case is triggered on average once every N runs, you're pretty sure to hit it pretty fast [17:46:57] dah right [17:47:32] okay, well what if we changed the cronjob to only run say, every 20 or so minutes? [17:48:02] that doesn't really change it [17:48:08] what you're saying is it keeps calling restart when older restarts haven't finished, right? [17:48:20] I don't know [17:49:16] well the point obviously is to have it restart when the service goes down. Earlier it was returning a 404, which webwatcher would detect and restart it, just like I would manually [17:49:25] so my guess is it's doing it too often [17:50:11] and I could make it kill whatever webservices are already running, for safe measure [17:51:25] Wouldn't it be easier to just move it over to the xtools project? [17:52:47] move what? the bash script? sorry not sure if I follow [17:53:13] the tool? [17:53:24] I mean, I'm not sure why the tool was down this time [17:53:45] xtools-ec? they were separated because -ec and -articleinfo are so popular and resource heavy [17:54:07] this way the whole suite doesn't go down when the more likely culprits go down [17:55:44] yes, and the new solution to the problem was 'moving everything off tools into the xtools project' [17:56:07] oh yes, I'm sorry, I thought you meant merge them back into the one tool on labs [17:56:28] yeah, the xtools project is the long-term goal but I think we're aiming for a rewrite first [17:56:47] perhaps we shouldn't wait and should just move over what we have for the time being [17:57:18] I would suggest that, because it might help debugging what's the underlying issue [17:57:25] because it's much easier to fiddle with server settings etc [18:00:11] thanks, well I'll try to round up the crew and make it happen. Fortunately the existing services don't go down too often now [18:00:51] in this case, I'm not sure how there was a 404 in the first place [18:01:25] (03PS5) 10Sitic: Add support for subdivided watchlists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/231789 (https://phabricator.wikimedia.org/T109188) [18:01:35] that either means xtools-ec threw the 404 itself, or the request was somehow mis-routed [18:01:54] yeah not sure [18:02:20] (03CR) 10Sitic: [C: 032 V: 032] Add support for subdivided watchlists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/231789 (https://phabricator.wikimedia.org/T109188) (owner: 10Sitic) [18:02:30] have to go, be back in a bit, thx! [18:20:27] 6Labs, 10Tool-Labs: SGE hosts down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554068 (10valhallasw) 3NEW [18:21:17] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554084 (10valhallasw) [18:21:52] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554068 (10valhallasw) [18:28:44] YuviPanda: question re-performance in labs ... we are trying to run some tests against data sets that are about 4-5x the memory of an XL instance. we end up at basically 50-70% IO wait on the xl instance. is there anything we can do about this? or is it just a world of spinning disks [18:30:25] ebernhardson: I'm not completely sure, but it might be possible to increase the memory available manually. andrewbogott would be able to do that [18:31:07] ebernhardson: YuviPanda is also working on 'bare bones' machines (i.e. actual servers that are provisioned in the same way as the virtual machines), but that's a more long term thing [18:32:28] valhallasw`cloud: thanks [18:32:34] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554148 (10scfc) I don't think that this is the cause. For example: ``` scfc@tools-bastion-01:~$ qhost -h tools-exec-1402 HOSTNAME ARCH NCPU... [18:32:41] ebernhardson: It’s possible to create special instance flavors with different memory profiles. But… you’re talking about needing 64Gb of ram? [18:33:20] andrewbogott: annoyingly, yes. It could be split between multiple instances though (but we are hitting the 50G quota in search project already) [18:34:02] are you doing performance testing or just wishing that things were faster? [18:34:35] andrewbogott: we have brought over the public elasticsearch indexes for dewiki and ptwiki, and are running queries against them in different ways to see how many results they bring back [18:35:06] andrewbogott: but the dewiki index is ~50GB, on the single 16GB machine queries take 15-30s each [18:35:31] it sits at 1.5MB-2.5MB of disk read per second (likely in random places, not contiguous) [18:35:42] sure, but… ‘load the whole dataset in memory’ is not an especially scalable way to handle large datasets. [18:36:06] andrewbogott: i would also accept raid'd ssds, but didn't think that was possible [18:36:29] (in prod we do both, 3.5TB of memory for the cluster and mirrored ssd's) [18:36:44] Anyway... I’m not sure if we have the excess capacity to allocate that kind of ram, but open a phab ticket with a specific request and I”ll research. It would help if you include info about which instances this would replace (so I know if RAM is being reclaimed at the same time) [18:37:55] andrewbogott: ok, thanks for checking. We'll also start poking around for real hardware if possible [18:42:13] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554208 (10valhallasw) [18:43:15] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554068 (10valhallasw) Sorry, I clarified it in the task description. It's caused by the discrepancy between the queue name and the hostname. The queues are named ....... [18:43:23] Can I do cors requests on labs? [18:43:33] it seems wgCrossSiteAJAXdomains doesn't allow wmflabs.org - not sure if there is a reason [18:46:55] Anything can appear in *.wmflabs.org [18:47:21] What wiki is that wgCrossSiteAJAXdomains from? [18:50:02] jdlrobson, ^ [18:50:40] I want to setup a labs instance and do a cors request to Wikipedia.org but it's refusing it because of the origin header not being allowed [18:55:42] jdlrobson, you want wikipedia.org to accept *.wmflabs.org as an origin of valid requests? [18:58:15] well i'm having to use something else to do this because of this restriction on get requests :-/ [18:58:40] but surely some labs instances do cors requests? [18:58:48] or are they just using jsonp? [18:59:13] what are you trying to do? [18:59:43] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554266 (10scfc) Deleting them doesn't seem to work: ``` scfc@tools-bastion-01:~$ qconf -dq mailq@tools-exec-1401 denied: cluster queue "mailq@tools-exec-1401" does n... [19:00:39] valhallasw`cloud: trying to write a service worker [19:01:08] ? [19:01:54] as in, you just need to read from the api? [19:02:11] yup [19:02:22] i have it working, but it only works if the domain is cors enabled [19:02:30] http://www.w3.org/TR/service-workers/ [19:02:42] https://developers.google.com/web/updates/2015/03/push-notificatons-on-the-open-web < i'm basically setting up a few of these [19:06:19] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554300 (10scfc) I `qconf -mhgrp \@general`, removed `.tools.` from `tools-exec-1401`'s host name and now: ``` scfc@tools-bastion-01:~$ qstat -f | grep 'exec-1401' ma... [19:07:50] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554305 (10scfc) (And if something could write up which command to use for which function, that would be much appreciated :-). Every time I try to tackle one of those... [19:08:03] huh. there's no Access-Control-Allow-Origin header at all, it seems. Okay, I think I just don't know enough of this to say much sensible [19:08:16] but I'd be inclined to solve it server side by (essentially) proxying [19:10:00] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554311 (10valhallasw) Yep, looks good to me. The only oddity left is ``` valhallasw@tools-bastion-01:~$ qstat -f -xml | less | grep '\.tools' webgrid-lig... [19:10:00] https://phabricator.wikimedia.org/T65808 seems this has been requested before [19:11:48] jdlrobson: ah. It's probably because private information is exposed over the api, even with GETs [19:11:59] e.g. username, watchlist [19:42:24] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554424 (10scfc) AFAIUI, that is to be expected as `tools-webgrid-lighttpd-1411` is not in `host_aliases`. In other words (and again: AFAIUI), if a host is in `host_a... [19:55:08] 6Labs, 10Tool-Labs: [tracking] Tool labs admin guides - https://phabricator.wikimedia.org/T104734#1554481 (10valhallasw) [19:55:10] 6Labs, 10Tool-Labs: 'new exec node' checklist - https://phabricator.wikimedia.org/T109417#1554478 (10valhallasw) 5Open>3Resolved a:3valhallasw https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin/new_exec_host [20:19:56] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554562 (10valhallasw) 5Open>3Resolved a:3valhallasw OK. Then I suggest to keep this one as .tools. and we can then slowly migrate towards a full .tools. environ... [20:30:17] 6Labs, 10Tool-Labs: tools-exec-1401/-catscan down (alarm/unknown) due to incorrect DNS/host_aliases - https://phabricator.wikimedia.org/T109605#1554626 (10valhallasw) and fwiw, I added a whole bunch of magic sge commands at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Administrative_tasks [21:06:14] andrewbogott: Tim reported "ECDSA host key for bastion-restricted.wmflabs.org has changed and you have requested strict checking." in another channel. bastion-01.bastion.eqiad.wmflabs seems uneffected [21:07:25] thanks, in theory this does what it did before and recovers but this is a controlled failover attempt unlike before :) [21:08:06] bd808: ok, I’m busy now but will try to catch up after this window [21:08:27] no worries. [21:11:24] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database, 5Patch-For-Review: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1554808 (10Dzahn) merged the change above. watched on silver. can now do this from tin: [tin:~] $ mysql -u wikiadmin -h silver.wiki... [21:14:41] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database, 5Patch-For-Review: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1554820 (10Dzahn) @jcrespo let's do `'wikiadmin'@'tin.eqiad.wmnet'` and `'wikiuser'@'tin.eqiad.wmnet'` for now? [21:19:02] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108: Evaluate a 'cluster solution' for use on Tool Labs - https://phabricator.wikimedia.org/T106475#1554836 (10Etune) Hi, I work on Kubernetes, and I'm new to the wikimedia environment. I see your requirement that "Proper user authentication / authorization that can tie... [21:20:30] 6Labs, 6operations, 10wikitech.wikimedia.org: Figure out what to do about maintenance scripts on silver/wikitech - https://phabricator.wikimedia.org/T107547#1554837 (10Dzahn) there is now a hole in the firewall to allow connections to mysql on silver from tin [21:21:54] papaul: hi [21:22:02] hi Daniel [21:22:04] so we are trying to get papaul on a labs instance via ssh [21:22:13] and we went through the ProxyCommand setup etc [21:22:19] the key is uploaded to wikitech, right [21:22:28] and a new instance was created [21:22:28] yes [21:22:33] yes [21:22:37] but he gets: Connection closed by 208.80.155.155 [21:22:42] Connection closed by 208.80.155.155 [21:22:42] ssh_exchange_identification: Connection closed by remote host [21:22:45] is that the race condition? [21:22:50] should he just try rebooting the instance? [21:24:15] can you connect to the bastion? [21:24:40] ah, there we go [21:24:44] so this is the ops bastion [21:24:58] Isn't the ops bastion broken at the moment or something? [21:25:01] maybe he needs to be added to that [21:25:06] duuh [21:25:08] Um, yeah [21:25:15] yes, what you said,, arg [21:25:18] You can't connect to the restricted bastion without being specifically authorised :) [21:26:37] papaul: can you "ssh bastion-eqiad.wmflabs.org" ? [21:26:53] let me try [21:26:54] Krenair: i'm unsure which one he is supposed to use then [21:27:14] andrewbogott: should papaul use ops bastion or regular bastion [21:27:26] I don't know the rules about ops connecting to non-ops bastions [21:27:28] mutante: ops bastion is required for ops [21:27:35] the other one won’t work for him [21:27:40] andrewbogott: do we need to add him somewhere ? [21:27:48] but it's currently down, right [21:27:56] mutante: no, it should just work. But, yes, I’m doing other things just now :) [21:28:00] What does it use to check this? The ops ldap group? [21:28:21] then the summary is it should work but doesn't [21:28:45] i can check the LDAP group [21:29:00] wonders again about the onboarding ticket [21:29:13] and whether it still makes sense to have separate bastions [21:29:15] I don't think papaul is in ldap/ops [21:29:17] without agent forwarding [21:29:38] mutante, I think we've had this discussion before :) Don't those restricted bastions allow all sorts of network access? [21:30:38] i don't know, what kind of network access [21:32:07] mutante i can not "ssh bastion-eqiad.wmflabs.org" [21:32:21] Permission denied (publickey). [21:32:43] papaul: yep, see what Andrew said above [21:33:51] mutante, can you connect to bastion-eqiad? [21:34:32] Krenair: connect, yes, but i'm not supposed to [21:34:45] So it does actually work for you [21:35:18] no [21:35:24] Connection closed by 208.80.155.129 [21:35:37] (yes, i unloaded the prod key before even trying) [21:35:43] labs is in a state of flux atm I wouldn't count any current behavior as authoritative [21:37:30] papaul: seems we have to just try again later [21:37:38] ok [21:37:46] thanks for your help [21:37:49] yw [21:38:11] well, we have the ssh config and all that already now [21:38:31] yep [21:39:51] question:how many instances i am allow to create [21:40:46] papaul: there is a quota per project, so it depends what "testlabs" quota is [21:40:52] it's not per user though, just per project [21:40:57] ok [21:41:12] you would notice when it tells you you can't create a new one :( [21:41:13] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database, 5Patch-For-Review: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1554926 (10Krenair) I'm not sure wikiuser is needed [21:41:14] :) [21:41:34] and then you could ask for the quota to be raised if needed [21:41:52] ok [21:57:59] Krenair: yes please. [22:01:30] YuviPanda: gut check… everything look good now? [22:02:02] andrewbogott: am on phone, so not sure :) Krenair ^ [22:02:14] yes mucho labs confirmation is good now [22:02:25] LGTM [22:06:08] 6Labs, 6operations: bastion-02.bastion.eqiad.wmflabs not restricted_from=(ops) like bastion-01 is - https://phabricator.wikimedia.org/T109641#1555032 (10Krenair) 3NEW a:3yuvipanda [22:08:56] YuviPanda: So, that was about 15 minutes of network downtime. Sorry to mess up our numbers :( We think we can do it a lot faster next time. [22:16:25] andrewbogott: :D so it's running on 1002 now? [22:16:31] and we can start upgrading things? [22:16:41] YuviPanda: yep! Thanks to chase working some router magic [22:16:53] woooo router magic! [22:18:32] 6Labs, 10Analytics, 10Labs-Infrastructure, 3Labs-Sprint-108, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1555114 (10yuvipanda) /me pokes @akosiaris again [22:30:39] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: Find missing wikitech apple touch icon (or remove reference) - https://phabricator.wikimedia.org/T102699#1555142 (10Krenair) a:3Krenair Okay, no response, removing instead [23:01:49] 6Labs, 10Tool-Labs: Request for installation of MongoDB package - https://phabricator.wikimedia.org/T108341#1555343 (10dhvanil) Hi! Sure, I understand. I had used mongodb for geoip lookups earlier for unrealated project (with some additional functionalities) so I just tweaked it to run the current tool. I ag... [23:02:48] 6Labs, 10Tool-Labs: Request for installation of MongoDB package - https://phabricator.wikimedia.org/T108341#1555349 (10dhvanil) 5Open>3Resolved [23:14:26] (03PS1) 10Jean-Frédéric: i18n related changes to API [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/232657 [23:28:56] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org, 5Patch-For-Review: Find missing wikitech apple touch icon (or remove reference) - https://phabricator.wikimedia.org/T102699#1555477 (10Krenair) 5Open>3Resolved [23:33:00] (03CR) 10Jean-Frédéric: "As said in the commit message, changes were made directly on the server. I’m trying to make sense of them in logical commits and get them " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/232657 (owner: 10Jean-Frédéric)