[00:27:27] !log deployment-prep Added gtirloni as a member per T207474 - I imagine he'll want to get in to look at shinken-related things [00:27:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [00:27:32] T207474: SSH auth to some Cloud VPS instances fails but root works - https://phabricator.wikimedia.org/T207474 [06:13:06] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/468921 (owner: 10L10n-bot) [06:13:09] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/468923 (owner: 10L10n-bot) [06:13:13] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/468924 (owner: 10L10n-bot) [06:37:38] (03PS1) 10Lokal Profil: Revert "Harvest whether an image is geolocated in the image table" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468928 (https://phabricator.wikimedia.org/T204036) [06:37:44] (03CR) 10jerkins-bot: [V: 04-1] Revert "Harvest whether an image is geolocated in the image table" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468928 (https://phabricator.wikimedia.org/T204036) (owner: 10Lokal Profil) [06:43:17] (03CR) 10Lokal Profil: "would you mind tagging it as WIP in gerrit? that way it doesn't show up in the normal review queue" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468581 (owner: 10Jean-Frédéric) [06:49:21] ragesoss: do you know about these apps? https://meta.wikimedia.org/wiki/Special:OAuthListConsumers?name=&publisher=Lyndonp80&stage=-1 [06:49:43] is the developer aware that mobile apps cannot store OAuth secrets? [07:49:43] (03CR) 10Jean-Frédéric: "> Patch Set 1:" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468581 (owner: 10Jean-Frédéric) [07:51:35] (03CR) 10Jean-Frédéric: "I’m fairly sure that this never ever worked. Harvesting images while hitting the database for every single one of them is a hopeless task " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468928 (https://phabricator.wikimedia.org/T204036) (owner: 10Lokal Profil) [07:52:32] (03CR) 10Jean-Frédéric: "Alternatively, we can leave the DB table as is with the geoloc column, and jsut always insert NULL in it..." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/468928 (https://phabricator.wikimedia.org/T204036) (owner: 10Lokal Profil) [10:26:11] !log admin change again in dmz_cidr in eqiad1: VMs will connect between them without NAT even when using floating IPs (T206261) [10:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:26:14] T206261: Routing RFC1918 private IP addresses to/from WMCS floating IPs - https://phabricator.wikimedia.org/T206261 [11:28:04] arturo: I’m getting [11:28:06] 11:09 icinga2-wm: PROBLEM - ping4 on ORES-worker01.experimental is WARNING: PING WARNING - DUPLICATES FOUND! Packet loss = 0%, RTA = 1.10 ms [11:28:33] And [11:28:35] 11:08 icinga2-wm: PROBLEM - ping4 on phab.wmflabs.org is WARNING: PING WARNING - DUPLICATES FOUND! Packet loss = 0%, RTA = 1.65 ms [11:29:34] that was 1h ago? [11:29:38] paladox: [11:29:41] Yep [11:29:44] Bst time [11:31:33] fixed already? [11:34:12] paladox: my understanding is that the issue was only briefly and should be already fixed [11:34:20] ah ok [11:34:21] yeh [11:34:24] seems fixed [15:46:24] (03CR) 10Gehel: [C: 04-1] "minor comments inline." (033 comments) [labs/private] - 10https://gerrit.wikimedia.org/r/468631 (https://phabricator.wikimedia.org/T206639) (owner: 10Mathew.onipe) [16:04:12] andrewbogott: I saw your email about instance migration, thanks! I'm also in the process of migrating away from Trusty. Can we combine these? I just create a new instance to start the migration process, can I delete that and create a new one in the new cluster? Or is the quota process the way to go to get access to do that? [16:04:23] *just created [16:04:56] Nettrom: Yep, I can enable your project in both regions so the new VMs you build are in the new region and don't need migrating. [16:05:04] hold on, I'll do that [16:05:16] Nettrom: project name? [16:05:27] andrewbogott: suggestbot [16:06:02] actually do you still need to create VMs in the old region? Maybe I can just disable creation there. [16:06:29] no, no need to be able to create VMs in the old region [16:06:40] I only have one VM, and it's running Trusty, it needs to go [16:07:51] (well, technically I have two VMs, but the newest one isn't really doing anything) [16:10:28] Nettrom: ok, if you look in Horizon now you should see VM creation disabled in eqiad and enabled in eqiad1-r. The other controls (e.g. delete instance) will still work in both regions. [16:13:18] andrewbogott: hm, seems to be the other way around… unfortunately, I gotta run to a meeting, so feel free to ignore this for a while, I'll follow up on it later :) [16:24:45] !log admin T206261 another update to dmz_cidr in eqiad1 [16:24:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:24:47] T206261: Routing RFC1918 private IP addresses to/from WMCS floating IPs - https://phabricator.wikimedia.org/T206261 [16:25:07] * paladox waits for "duplicates" again :) [16:27:46] paladox: not this time, I hope :-) [16:27:52] hehe [16:28:47] I should create a puppet control to instruct neutron to only have one server active at a time or something like that, but not sure how neutron will take that... [16:29:27] so far so good :) [16:29:47] https://gerrit-icinga.wmflabs.org/dashboard#!/monitoring/service/show?host=ORES-redis02.experimental&service=ssh hasen't recovered [16:30:03] so i guess ores is still seeing neutron as a public ip [16:30:33] username: guest, password: guest [16:31:11] * arturo checking [16:31:46] paladox: could you manually check to see what happens? [16:32:01] not sure what you mean? [16:32:18] I belive it's because nrpe is seeing neutron as a public ip [16:32:30] the new ip was added to the whitelist. [16:32:59] let me check one thing [16:33:28] arturo https://wikitech.wikimedia.org/w/index.php?title=Hiera%3AOres&type=revision&diff=1803480&oldid=1801745 [16:35:38] what I want to do paladox is to manually try the same check that icinga is doing [16:35:44] ok [16:36:41] https://github.com/wikimedia/labs-icinga2/blob/master/templates/services.conf.erb#L26 [16:37:12] nvm [16:37:18] about the link wrong service [16:37:27] https://github.com/wikimedia/labs-icinga2/blob/master/templates/services.conf.erb#L50 [16:38:14] /usr/lib/nagios/plugins/check_ssh -H ores-redis-02.ores.eqiad.wmflabs [16:38:36] paladox: from the gerrit-mysql hsot? [16:38:38] host* [16:38:40] yup [16:38:50] it works with /usr/lib/nagios/plugins/check_ssh -H gerrit-test.git.eqiad.wmflabs [16:38:54] (which is in neutron) [16:38:58] but not /usr/lib/nagios/plugins/check_ssh -H ores-redis-02.ores.eqiad.wmflabs [16:39:04] root@gerrit-mysql:/home/paladox# /usr/lib/nagios/plugins/check_ssh -H ores-redis-02.ores.eqiad.wmflabs [16:39:04] CRITICAL - Socket timeout after 10 seconds [16:48:26] paladox: it seems to me there are a firewall/filter somewhere that prevents the packet from reaching the ores-redis-02 machine [16:48:35] oh [16:48:36] could you please check the security group/port security? [16:48:47] Im not in the ores project [16:50:50] ok [16:50:59] * arturo off [17:00:15] arturo it works! [17:00:25] awight added the nagios group to ores-redis [17:14:21] (03PS1) 10Jforrester: Move Infrastructure feeds to new channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/469035 [17:14:22] (03PS1) 10Jforrester: Add maps items to Infrastructure team channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/469036 [17:16:34] (03CR) 10Jforrester: "Someone needs to talk "to freenode" (not sure who the usual contacts for this are)." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/469035 (owner: 10Jforrester) [18:09:21] !log suggestbot Shut down and deleted suggestbot-01 instance from eqiad [18:09:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Suggestbot/SAL [18:09:54] andrewbogott: thanks for getting me set up to migrate to the new region! logged in and found that I can no longer create instances in the old region [18:10:07] looking forward to giving the new region a go :) [18:10:07] Nettrom: cool [20:55:06] !log integration migrated integration-slave-docker-1017 to eqiad1-r [20:55:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL [20:55:18] !log integration migrating integration-slave-docker-1033 to eqiad1-r [20:55:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL [21:20:59] !log toolsbeta launched a stretch/sonofgridengine master server [21:21:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [23:22:09] !log migrating integration-slave-jessie-1003 and integration-slave-jessie-1004 to eqiad1-r [23:22:10] andrewbogott: Unknown project "migrating" [23:22:17] !log integration migrating integration-slave-jessie-1003 and integration-slave-jessie-1004 to eqiad1-r [23:22:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL