[11:21:12] <wm-bot>	 !log tools.flickrdash <samwilson> Updating to version 0.6.0
[11:21:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.flickrdash/SAL
[13:39:59] <arturo>	 !log tools icinga downtime toolschecker for 1h for replacing SSL cert T235252
[13:40:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:40:04] <stashbot>	 T235252: Toolforge: SSL support for new domain toolforge.org - https://phabricator.wikimedia.org/T235252
[13:48:01] <arturo>	 !log tools replacing SSL cert in tools-proxy-x server (live-hacking https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679 first for testing) T235252
[13:48:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:48:06] <stashbot>	 T235252: Toolforge: SSL support for new domain toolforge.org - https://phabricator.wikimedia.org/T235252
[13:53:22] <arturo>	 !log tools replacing SSL cert in tools-proxy-x server apparently OK (merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/545679) T235252
[13:53:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:54:20] <stashbot>	 T235252: Toolforge: SSL support for new domain toolforge.org - https://phabricator.wikimedia.org/T235252
[15:01:09] <wm-bot>	  Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts:  - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[15:50:54] <wm-bot>	  Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts:  - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[16:15:41] <mmecor>	 hi!
[16:15:42] <mmecor>	 does anyone know where the thumbs URLs are stored in commons database or they generated from the filename?
[16:28:13] <Lucas_WMDE>	 the hex part at the beginning of the thumb URL is the beginning of the MD5 hash of the file name, I believe
[16:32:23] <mmecor>	 getting to that is not difficult 
[16:32:29] <mmecor>	 i can also use the special redirect with the filename
[16:32:38] <mmecor>	 but i don't know the sizes
[16:32:47] <mmecor>	 in which the image is available
[16:35:48] <Lucas_WMDE>	 could be any number of sizes
[16:35:53] <Lucas_WMDE>	 Tgr posted a good overview here: https://discourse-mediawiki.wmflabs.org/t/embedding-wikimedia-commons-images-in-tools-with-multiple-resolutions-srcset/1501/2?u=lucaswerkmeisterwmde
[16:40:16] <mmecor>	 Thank you! :)
[16:51:31] <bblack>	 hey, our network monitoring is showing some huge bursts of traffic from Cloudflare towards Cloud VPS ( nat.openstack.eqiad1.wikimediacloud.org )
[16:51:48] <bblack>	 is this part of some known project/effort that's fronting some VPN service via Cloudflare?
[16:51:59] <bblack>	 err sorry s/VPN/VPS/ above
[16:52:19] <bd808>	 bblack: not that I am aware of, no.
[16:54:38] <bblack>	 looking at the details of the report, it actually looks more like...
[16:55:13] <bblack>	 like something in Cloud VPS, whose outbound connections would NAT out through the above nat.openstack address, pulled a crapload of data down from some Cloudflare-hosted HTTPS site, very very quickly.
[16:55:37] <bblack>	 (like, 647Mbps of traffic, enough to trip some alarms)
[16:56:20] <bd808>	 bblack: that is certainly something that could happen, yes. There are a lot of tools that are used for importing media or datasets to various wikis.
[16:56:21] <bblack>	 but that could be anything, and the user/service/thing doing it may not even be aware of the specifics above (that whatever site they're pulling tons of data from happens to be hosted by Cloudflare's CDN)
[16:56:48] <bblack>	 anyways, I guess there's nothing to do here, just making sure you're aware in case
[16:57:16] <bd808>	 bblack: arturo or jeh might be able to help figure out what the NAT'ed client is if you think that is worthwhile
[16:57:45] <bblack>	 right now so far it's probably not worthwhile
[16:57:57] <bblack>	 unless it grows so big that it starts saturating our links or something, but we're not there yet :)
[16:58:02] <arturo>	 can I see that report you mention bblack ?
[16:58:40] <bblack>	 arturo: I'm not sure if you're on the noc@ alias, you can check your gmail for the search string "fastnetmon"
[16:59:15] <bblack>	 I can forward to you otherwise
[17:00:02] <arturo>	 this is a graph of traffic in openstack
[17:00:05] <arturo>	 https://grafana.wikimedia.org/d/000000579/wmcs-openstack-eqiad1?refresh=30s&panelId=36&fullscreen&orgId=1
[17:00:25] <arturo>	 it shows nothing weird in the last few days
[17:01:13] <bblack>	 yeah unless it's a sampling/averaging issue (our fastnetmon may have tripped on a very short but very high-bandwidth spike, and it's averaged down and dissappears in that view)
[17:02:07] <bblack>	 anyways, forwarded emails to you in case you want to peek
[17:02:14] * arturo nods
[17:02:16] <bblack>	 but so far it hasn't caused a real problem
[17:02:22] <arturo>	 BTW yes I get noc@
[17:02:39] <bblack>	 ah ok :)
[17:06:01] <arturo>	 I dont think we have any infra to see traffic per project
[17:06:13] <arturo>	 inside CloudVPS
[17:06:37] <arturo>	 so I'm not sure what else to check
[17:06:57] <arturo>	 what bd808 makes sense
[17:08:05] <arturo>	 we usually have problems with people over using the outbound connections and hitting ratelimits and stuff, recently happened with youtube
[17:08:28] <arturo>	 what bd808 said*
[19:13:58] <Wurgl>	 ffmpeg is running. Can't this guy use /tmp for the files to convert? CPU is not the problem, it is I/O! It is really not funny to wait 10 seconds and more for some simple command
[19:14:03] <Wurgl>	 </rant>
[19:16:50] <zhuyifei1999_>	 !log tools.faebot SIGSTOP ffmpeg process running on tools-sgebastion-07
[19:16:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.faebot/SAL
[19:19:31] <zhuyifei1999_>	 I froze it. He can unfreeze, but meh, he could rerun...
[19:20:05] <zhuyifei1999_>	 actually, I'll `write` to him
[19:20:34] <Wurgl>	 Seriously: is using /tmp possible?
[19:23:41] <zhuyifei1999_>	 yes
[19:23:49] <zhuyifei1999_>	 unless /tmp has too little space left
[19:24:10] <zhuyifei1999_>	 it should have 3 gigabytes left
[19:24:18] <Wurgl>	 it has 3 gig
[19:24:34] <zhuyifei1999_>	 I `write` to him saying to use /tmp
[19:24:42] <bd808>	 thanks for dealing with that zhuyifei1999_
[19:24:50] <zhuyifei1999_>	 np
[19:25:20] <bd808>	 I kind of wonder if they created the youtube rate limit problem in the first place that now has them abusing the bastion
[19:25:50] <bd808>	 toolforge is not a great worker pool for downloading huge external files
[19:27:19] <zhuyifei1999_>	 I dunno what triggered the rate limit in the first place. once upon a time v2c was flooded and it didn't trigger rate limits
[19:29:30] <Wurgl>	 Use TOR to download from youtube … okay, that must be installed
[19:29:33] <Wurgl>	 *g*
[19:41:03] <zhuyifei1999_>	 I wonder if youtube blocks TOR exit nodes like we do
[19:42:36] <wm-bot>	 !log tools.integraality <jeanfred> Deploy latest from Git master: 289a41b, 02a1a4f, e8d4363 (T224226)
[19:42:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[19:42:50] <bd808>	 We only block editing from TOR, not reading. I would be pretty surprised if YouTube blocked viewing via TOR
[19:43:16] <bd808>	 <<my thoughts on the value of the TOR edit block are censored>>
[19:46:04] <Wurgl>	 zhuyifei1999_: Hmm torify youtube-dl <file> crashes here hmm
[19:49:46] * zhuyifei1999_ tries
[19:50:39] <Wurgl>	 zhuyifei1999_: without torify it worked with that testfile, however it crashed in both cases with another 
[19:51:00] <Wurgl>	 <-- using opensuse 15.0
[19:53:51] <zhuyifei1999_>	 torify youtube-dl works for me
[19:54:13] <zhuyifei1999_>	 <-- Gentoo. I'm nuts, I know :)
[19:54:31] <Wurgl>	 A long time ago, i used gentoo.
[19:54:32] <zhuyifei1999_>	 it makes youtube-dl much slower though
[19:55:23] <Wurgl>	 But i bought this machine with preinstalled suse (not sure if suse 10 or 11)
[19:58:08] <Krenair>	 subbu, re https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/545702/ - if it's not merged, cherry-pick it on the deployment-puppetmaster host
[19:59:46] <zhuyifei1999_>	 Wurgl: nice. I always had to install linux myself
[20:00:01] <zhuyifei1999_>	 :/
[20:00:28] <zhuyifei1999_>	 though, I don't think gentoo would be preinstalled. it would usually be ubuntu
[20:02:22] <Wurgl>	 Since I am living in germany and Suse has its roots in Nuremberg …
[20:04:36] <Wurgl>	 AND! That was in march 2009 …
[20:17:19] <zhuyifei1999_>	 nice
[21:35:51] <wm-bot>	 !log tools.stewardbots <maurelio> Rebooting StewardBot and SULWatcher
[21:35:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[21:44:33] <hip>	 anybody around to field a quick SSH config question?
[21:46:40] <Krenair>	 hi hip
[21:46:40] <Krenair>	 sure
[21:47:07] <hip>	 Hi there, just trying to determine the correct bastion host I should be connecting to, I haven't had to use that part of my ssh config for about a year and the documentation is a bit conflicting
[21:48:43] <Krenair>	 hip, the main bastion server for Cloud VPS is bastion.wmflabs.org.
[21:49:30] <Krenair>	 Tools project members can probably proxy through login.tools.wmflabs.org if they like.
[21:49:44] <Krenair>	 Tools users generally SSH directly to login.tools.wmflabs.org
[21:49:47] <Krenair>	 (I assume)
[21:50:01] <hip>	 hmm
[21:50:37] <hip>	 I'm an engineer in Product Infrastructure, so I have a developer account although I don't know what access groups I'm in or anything
[21:50:57] <hip>	 essentially trying to connect to a project instance, the connection is getting closed by the bastion host
[21:51:01] <Krenair>	 I don't know what Product Infrastructure is but okay
[21:51:10] <hip>	 sorry it's one of the teams in the Product department of WMF?
[21:51:11] <Krenair>	 what instance?
[21:51:25] <hip>	 deployment-logstash2.deployment-prep.eqiad.wmflabs
[21:51:28] <Krenair>	 okay
[21:51:31] <Wurgl>	 on bastion type "id" and you see the tools
[21:51:50] <Krenair>	 Don't worry about anything to do with the tools project in this case :)
[21:52:10] <Krenair>	 hip, what's your username?
[21:52:44] <hip>	 the shell account is 'jdl', the username is 'Jason Linehan'
[21:53:22] <Krenair>	 mm I don't see any attempts for that username in logstash2's auth.log
[21:53:40] <Krenair>	 maybe the bastion host is closing the connection before you can even get that far
[21:53:46] <hip>	 That's what I'm thinking
[21:53:59] <Krenair>	 can you pastebin the output of ssh with -vvv ?
[21:54:39] <hip>	 https://pastebin.com/yxCEcmia
[21:54:53] <Krenair>	 actually I just checked and you're not in the deployment-prep project, so even if you could get into the bastion you'd be rejected by deployment-logstash2
[21:55:18] <Krenair>	 huh
[21:55:24] <Krenair>	 what if you try to ssh to bastion.wmflabs.org directly?
[21:56:10] <hip>	 https://pastebin.com/HHLfGLUx
[21:56:27] <hip>	 yeah it's a bit strange, it seems like it authenticates, but then closes the connection
[21:56:30] <Krenair>	 Interesting.
[21:56:34] <Krenair>	 debug1: Authentication succeeded (publickey).
[21:56:37] <Krenair>	 Authenticated to bastion.wmflabs.org ([185.15.56.13]:22).
[21:56:53] <jeh>	 looks like your ssh key is not correct `Failed publickey for jdl`
[21:57:09] <hip>	 okay, now is this the key that gets set on-wiki?
[21:57:12] <Krenair>	 jeh, that's from the bastion auth logs?
[21:57:21] <jeh>	 Krenair: yes
[21:57:27] <Krenair>	 this'll be the key you set in wikitech preferences yeah
[21:58:13] <hip>	 I tried to re-set that, afaict it's what corresponds to the key i'm offering. Is there any weirdness to be aware of? I'll try again
[21:58:26] <Krenair>	 jeh, that's... interesting. is that the latest log relating to jdl or an older one?
[21:59:33] <jeh>	 Krenair: it's all of them
[21:59:40] <Krenair>	 bizarre
[21:59:41] <jeh>	 for today at least
[21:59:59] <Krenair>	 client thinks its successfully authenticated but server logs disagree?
[22:00:15] * jeh looking
[22:01:05] <Krenair>	 hip, are you on a trustworthy network?
[22:01:33] <hip>	 yeah, well I'm on a home network
[22:01:47] <Krenair>	 and your ISP isn't likely to be doing anything dodgy like intercepting SSH
[22:02:38] <Krenair>	 hold on a minute
[22:02:47] <Krenair>	 don't people need to be members of the bastion project to log in?
[22:03:15] <Krenair>	 krenair@bastion-eqiad1-01:~$ ldapsearch -x member=uid=jdl,ou=people,dc=wikimedia,dc=org | grep dn:
[22:03:15] <Krenair>	 dn: cn=wmf,ou=groups,dc=wikimedia,dc=org
[22:03:15] <Krenair>	 krenair@bastion-eqiad1-01:~$ 
[22:03:26] <Krenair>	 When I run that on myself I get a project-bastion entry.
[22:03:28] <jeh>	 I see 'Authentication succeeded (publickey)' in the client logs, but nothing in the auth logs match it
[22:04:03] <Krenair>	 This stuff was granted automatically as soon as someone is added to a different project right? So this might all resolve once hip is added to deployment-prep?
[22:05:25] <jeh>	 sounds right, he'll need to be added to that project anyways.
[22:06:30] <Krenair>	 https://www.mediawiki.org/wiki/Wikimedia_Product/Wikimedia_Product_Infrastructure_team shows James is on this team
[22:06:59] <hip>	 Krenair: that's mostly because we haven't updated the page in a while
[22:07:01] <Krenair>	 James_F, please can you confirm this is the real Jason and is okay to have deployment-prep access?
[22:07:38] <hip>	 Krenair: but James and I know eachother (Hi James)
[22:09:24] <jeh>	 Krenair: you're correct, `Everyone who is in a project has access to the bastion machines` https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_Cloud_VPS_instances
[22:09:45] <Krenair>	 I think there's some magic that notices people got added to their first project and adds them to project-bastion.
[22:09:49] <Krenair>	 But I forgot where we hid it.
[22:10:02] <hip>	 Krenair, that might do it. I don't think I'm on any projects so far.
[22:10:08] <Krenair>	 yeah you're not yet
[22:10:08] <hip>	 Sounds plausible.
[22:17:02] <Krenair>	 hip, James might be AFK or something, alternatively perhaps you could edit a wiki from https://meta.wikimedia.org/wiki/Special:CentralAuth/JLinehan_(WMF) saying "my username is jdl and I'd like deployment-prep access"?
[22:17:31] <Krenair>	 like somewhere under your user space or something
[22:17:41] <hip>	 Krenair, sure just a second
[22:19:31] <hip>	 Krenair: https://www.mediawiki.org/wiki/Wikimedia_Product/Analytics_Infrastructure
[22:19:34] <hip>	 see bottom of page
[22:19:51] <Krenair>	 https://www.mediawiki.org/w/index.php?title=Wikimedia_Product/Analytics_Infrastructure&diff=3487107&oldid=3320194&diffmode=source
[22:19:54] <Krenair>	 that's from an IP
[22:20:04] <hip>	 oh you're right, whoops hang on
[22:21:06] <hip>	 try again
[22:21:19] <jeh>	 hip could you make a phabricator task for this please?
[22:21:22] <Krenair>	 ok
[22:21:31] <Krenair>	 don't think it needs a phab  task
[22:21:41] <hip>	 I don't mind, it's up to you guys
[22:22:20] <Krenair>	 alright
[22:22:21] <Krenair>	 hip, try now
[22:23:17] <Krenair>	 You are in project-bastion and project-deployment-prep now
[22:23:19] <James_F>	 Krenair: Sorry, yes, hip is Jason.
[22:23:23] <Krenair>	 so that worked
[22:23:34] <Krenair>	 Still don't know where the bastion one magically appears from but still
[22:23:37] <hip>	 James_F, thanks!
[22:23:38] <Krenair>	 thanks James :)
[22:23:56] <James_F>	 Typical me, too late to be useful but still getting thanked by lovely colleagues. ;-)
[22:24:00] <Krenair>	 :D
[22:24:11] <hip>	 Krenair, yes to confirm it works fine now, thanks all, jeh too
[22:43:22] <jeh>	 !log admin reboot cloud-bootstrapvz-stretch to resolve bad bootstrapvz build
[22:43:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[23:49:13] <bd808>	 Krenair: the membership in the bastion project is added by a hook in Keystone that fires when people are added/removed from Cloud VPS projects. When that action is add then it checks to see if you are a bastion member already and adds if not. On remove it checks to see if you are still a member of any project in addition to bastion and removes if not.
[23:49:54] <Krenair>	 bd808, sounds logical. any idea where the source lives?
[23:50:18] <bd808>	 I think wmfkeystonehooks.py in ops/puppet
[23:52:24] <Krenair>	 I guess the reason I couldn't find it was it doesn't operate directly on LDAP, it's project membership in keystone which writes to project-%s LDAP?
[23:53:19] <bd808>	 https://github.com/wikimedia/puppet/blob/production/modules/openstack/files/mitaka/keystone/wmfkeystonehooks/wmfkeystonehooks.py#L284
[23:53:43] <bd808>	 hmmm.. that's for a new project into ldap.
[23:53:49] <bd808>	 there must be another sync too
[23:54:19] <bd808>	 ah, yes -- https://github.com/wikimedia/puppet/blob/production/modules/openstack/files/mitaka/keystone/wmfkeystonehooks/wmfkeystonehooks.py#L122-L125
[23:56:02] <bd808>	 the ldap group membership is still used by NSS related things, but is not used by OpenStack itself anymore. Which leads to these "fun" hidden connections between the Keystone authz data in mysql and the LDAP directory
[23:58:30] <Krenair>	 cool