[00:29:41] !log tools migrating tools-exec-1411 and tools-exec-1410 off of cloudvirt1017 [00:29:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [06:53:42] (03CR) 10Nikerabbit: [V: 032] Localisation updates from https://translatewiki.net. [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/468217 (owner: 10L10n-bot) [07:36:08] (03CR) 10Urbanecm: "recheck" [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [07:36:54] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [07:40:47] (03PS15) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [07:41:52] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [07:46:17] (03PS16) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [07:46:49] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [10:14:25] !log admin we have new virt servers in the eqiad1 deployment since past week and this week: cloudvirt1018, cloudvirt1023, cloudvirt1024 [10:14:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:16:27] !log admin change in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs (T206261) [11:16:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:16:30] T206261: Routing RFC1918 private IP addresses to/from WMCS floating IPs - https://phabricator.wikimedia.org/T206261 [11:33:32] arturo: nice :) [11:41:37] arturo: hi, I’m not sure if that change is causing this intermittent problem: [11:41:39] 12:33 icinga2-wm: PROBLEM - ping4 on phab.wmflabs.org is WARNING: PING WARNING - DUPLICATES FOUND! Packet loss = 0%, RTA = 1.53 ms [11:42:19] what is that check doing paladox ? from where to where is the ping? [11:42:59] It is in gerrit-mysql and the check is ping (running icinga2) [11:43:25] It is checking phab.wmflabs.org (which connects to phabricator) [11:49:40] paladox: I mean the machine name [11:50:18] oh, the machine running the check is gerrit-mysql [11:51:14] gerrit-mysql.git.eqiad.wmflabs [11:51:23] I see now the issue [11:52:08] Ah :) [11:56:56] reverting patch... [12:02:47] !log admin revert change in dmz_cidr in eqiad1 for now (T206261) [12:02:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:02:50] T206261: Routing RFC1918 private IP addresses to/from WMCS floating IPs - https://phabricator.wikimedia.org/T206261 [13:59:09] (03PS1) 10Jean-Frédéric: [WIP] Fix statisticsct [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468581 [14:00:04] (03CR) 10Jean-Frédéric: "I spent some time a few weeks ago trying to fix this ; but the more I worked on it the more it seemed somehow hopeless… Stashing this just" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468581 (owner: 10Jean-Frédéric) [14:00:58] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Fix statisticsct [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468581 (owner: 10Jean-Frédéric) [14:17:56] !log tools moving tools-clushmaster-01 to labvirt1004 [14:17:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:54:11] chicocvenancio, addshore, is the 'general-k8s' project something that supports external users or just a sandbox? I'm wondering if it needs advance notice about region migration or if I can just do it today at my convenience. [14:54:42] harej: you are also an admin in that project as is 'albe' who seems not to be on IRC [15:05:43] !log glampipe moving the Glampipe VM to labvirt1010 and shutting down as that project is unclaimed on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge [15:05:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Glampipe/SAL [15:06:31] andrewbogott: I haven't looked at that project in a few months, so I'm not the best to speak about it [15:06:53] there would be external users, but I don't think it is ready [15:06:55] 'k [15:07:00] It's possible of course that no one has :) [15:07:58] indeed [15:26:53] andrewbogott: cna you confirm whether Monday or Tues next week. [15:26:56] er [15:27:00] https://phabricator.wikimedia.org/T207387 [15:27:02] is resolved? [15:29:01] paravoid: I'll look soon — thanks for working on it [15:29:44] yw [15:29:59] ftr, I don't think this is our responsibility, but I'm just trying to be helpful [15:30:24] I think whoever maintains each project should be responsible for that kind of thing [15:31:02] at least for initial troubleshooting and something like this (a package backport) -- if there is a bigger amount of work or something foundational we can always discuss [15:31:21] I don't agree that the trusty EOL timeline has anything to do with that, tbh [15:31:58] our monitoring servers were upgraded from trusty->jessie in 2015, the fact that e.g. Shinken remained behind was tech debt on whoever maintains it [15:32:14] and taking on tech debt is fine sometimes! depends on one's priorities :) [15:32:48] but we can't be the ones paying another team's tech debt, 3 years after the fact [15:33:33] https://phabricator.wikimedia.org/T205091 was another case of that for Shinken specifically, these are distractions for everyone [15:34:29] and the general concept applies more broadly I think, and especially when you have WMCS projects operated by different people/teams/etc., each with their own priorities, and thus impossible to tie all together [15:45:00] paravoid: puppet is running on -01 again so I think we're good for now. Hopefully we'll get that off of Trusty soon. [15:45:37] great [17:57:23] @andrewbogott, nothing real is running there yet, so migrate as you wish [18:27:33] addshore: thanks, I might do some of that today then [18:31:05] !log general-k8s migrating VMs to eqiad1 [18:31:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:General-k8s/SAL [18:45:04] andrewbogott you can also migrate planet if you want too. [18:45:20] great! [18:47:04] !log planet migrating project to eqiad1 [18:47:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL [19:18:14] (03PS1) 10Mathew.onipe: maps: remove osmupdater and osmimporter hiera passwords [labs/private] - 10https://gerrit.wikimedia.org/r/468631 (https://phabricator.wikimedia.org/T206639) [19:22:01] Hey, I am trying to use the docker container for toollabs (docker-registry.tools.wmflabs.org/toollabs-php-web) but it's php 5.6. I need php7.2 - is there an alternative container ? [19:25:00] bd808: ^ do you have any ideas on this? Trying to build our dev environment for our cloud tool. It's using php7.2, so I thought we probably have a container for that? [19:25:55] mooeypoo: there is a PHP 7.2 container ... let me find the label [19:26:01] woot [19:26:35] Is there a place where I can see all available containers related to tool labs? I was trying the docker registry but it's not as easy to find by category or something there [19:27:16] "toollabs-php72-web" is the 7.2 image [19:27:25] I just looked at https://docker-registry.tools.wmflabs.org/v2/_catalog [19:28:14] sweet [19:28:18] oh! _catalog [19:28:21] got it. Thanks! [19:29:32] Would be helpful to have a more... friendly list somewhere, though. [19:29:38] mooeypoo: if you want to use it actually on Toolforge its `webservice --backend=kubernetes php7.2 start|stop|shell|...` -- https://lists.wikimedia.org/pipermail/cloud/2018-August/000367.html [19:30:24] I think we already are (?) I was working on a local dev env to be as close to the cloud one we already have as possible. Plus it's SUPER helpful that the cloud 'web' container includes the web service, php AND composer [19:30:40] mooeypoo: yeah, I have a note laying around somewhere about building a tool to make a pretty display of the catalog. I looked around and there are a few frontends that might be useable [19:30:43] otherwise I ran in circles trying to set these containers separately and run composer after [19:31:22] * bd808 should make sure there is a ticket for that in phab [19:35:32] andrewbogott what does "detach interface" do? [19:35:54] paladox: I'm surprised you can see that option… it will most likely break your VM forever :) [19:35:56] where is it? [19:36:04] I see it in the ui [19:36:08] * paladox screenshots it [19:36:55] andrewbogott https://phabricator.wikimedia.org/F26637564 and https://phabricator.wikimedia.org/F26637568 [19:39:52] !log striker migrating project to eqiad1 [19:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [20:24:39] !log shinken temporarily increased project quota (T204562) [20:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [20:24:42] T204562: cloudvps: shinken project trusty deprecation - https://phabricator.wikimedia.org/T204562 [21:29:29] gtirloni, why have you increased the quota in shinken? [21:32:10] Krenair: so I could create a 3rd (!) VM to test a few things without breaking everything else.. It seems version 2.4.3 fixes the bugs we're facing (it recovers fine from sporadic timeouts in the pings), but it means we have to backport the package and upgrade it.. since I'm down that path, I thought it would make sense to test on Stretch [21:35:12] s/It seems version 2.4.3 fixes/Version 2.4.3 fixes/ -- tested successfully [21:40:43] (03CR) 10Urbanecm: "recheck" [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [21:41:05] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [21:42:48] (03PS17) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [21:43:41] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [21:47:53] (03PS18) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [21:48:50] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [21:49:43] (03PS19) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [21:50:43] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [21:55:21] (03PS20) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [22:01:14] (03PS21) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [22:05:18] (03PS22) 10Urbanecm: [WIP] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [22:07:28] (03PS1) 10Urbanecm: Include build status badge in README [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/468681 [22:08:32] (03CR) 10Urbanecm: [C: 032] Include build status badge in README [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/468681 (owner: 10Urbanecm) [22:08:49] (03Merged) 10jenkins-bot: Include build status badge in README [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/468681 (owner: 10Urbanecm) [22:11:06] (03PS23) 10Urbanecm: Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [22:13:03] (03CR) 10Urbanecm: [C: 032] Use SQLAlchemy instead of pure SQL queries [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467949 (https://phabricator.wikimedia.org/T207228) (owner: 10Urbanecm) [22:13:22] (03Merged) 10jenkins-bot: Use SQLAlchemy instead of pure SQL queries [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467949 (https://phabricator.wikimedia.org/T207228) (owner: 10Urbanecm) [22:13:58] (03CR) 10Urbanecm: [C: 032] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [22:14:00] (03CR) 10jerkins-bot: [V: 04-1] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [22:15:11] (03PS24) 10Urbanecm: Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 [22:15:40] (03CR) 10Urbanecm: [C: 032] Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [22:16:54] (03Merged) 10jenkins-bot: Add basic tests to Wikinity [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/467823 (owner: 10Urbanecm) [22:33:03] SMalyshev: OK if I delete that VM in an hour or two? Or do you need it for more testing? [22:40:35] !help - is here an ETA for fixing the replag on wikidatawiki.labsdb - see https://tools.wmflabs.org/replag/ ? [22:40:36] tagishsimon: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [22:53:06] tagishsimon: Wow! That's pretty bad. I don't have a timeline. That would involve the DBAs, but making a ticket might be worth it in case nobody has noticed? Let me see if there is already one out there [22:54:18] thx bstorm_ ... I couldn't find a ticket based on a search for wikidatawiki.labsdb ... happy to make one (or read one you make if you're more familiar) [22:55:05] In particular Petscan queries involving wikidata are suffering. [22:57:47] Why don't you make the ticket? I cannot find any reference to any schema changes or anything that would stop replication, but I don't have the whole environment on my radar. With a ticket out there, someone else might be able to provide better info. I know the database folks are well into their evenings/weekends. [22:58:41] will do [22:58:44] Let me know the number and I'll subscribe and see if I can make sure the right people get tagged or whatever. [23:01:49] https://phabricator.wikimedia.org/T207524 [23:03:23] thanks again, bstorm_ [23:05:08] !help My instances seem to have disappeared from Horizon [23:05:08] tom29739: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [23:05:21] tom29739: check to see what region is selected in the project pulldown [23:05:29] probably your VMs are in 'eqiad' unless you've been notified of a move [23:06:07] I see them now [23:06:10] Thanks [23:06:21] np — it'll be less confusing after we finish this transition