[00:00:46] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3321776 (10kaldari) @Samwilson: FWIW, the counts in the `user_editcount` field should now stay closer to the XTools-specific edit counts, thanks to T163966. Are you planning on u... [00:02:28] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:04:17] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3321781 (10Samwilson) @kaldari Cool. Yeah, it's currently using the `globaluserinfo` API call, which uses `user_editcount` doesn't it? — but I'll switch it to use the DB directly... [00:29:28] Herp derp. [00:29:33] Was kicking really necessary? [00:33:43] Especially with no warning, explanation, or anything else like that... [00:37:31] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [00:40:55] Yeah. [00:48:04] It was a redirect, apologies it comes with a message that says you've been kicked! [00:48:15] the ways of irc i guess [00:52:45] I mean, it did kick/forcibly remove everyone from the room. [00:53:15] And at least in my client, a kick doesn't result in a re-join, for better or worse, so I never hit the redirect. [00:53:23] It's not a huge deal. [00:53:48] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10Tracking: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3321906 (10bd808) [00:53:50] 10Labs, 10Tool-Labs: Add dependencies for Postgresql to Kubernetes container - https://phabricator.wikimedia.org/T161266#3321905 (10bd808) 05Open>03stalled [00:53:56] Though I am pretty amused that we're abandoning "labs" in favor of "cloud" for clarity(???). [00:54:02] Yvette: yeah sorry about that, we did send out some announcements in anticipation of this [00:54:15] Ah, probably missed them. [00:54:22] I'm still not subscribed to labs-l. [00:54:31] Is that becoming cloud-l?! [00:54:55] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10cloud-services-team (Kanban): Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3321907 (10bd808) @Xelgen, can you run the tool you would like to build on Grid Engine rather than Kubernetes? It s... [00:55:02] Yvette: yes it will be, it is currently labs-l [00:55:04] https://phabricator.wikimedia.org/phame/blog/view/5/ [00:55:06] Fair play, it's right here: https://lists.wikimedia.org/pipermail/labs-l/2017-June/005012.html [00:55:20] we publish updates here on this phab blog too, as of few days ago [00:55:27] Oh my, unicorns, too. [00:55:40] Thanks for the link, I subscribed. [00:55:54] Yvette: yw :) [00:56:07] Only 43,000 unread Phabricator e-mails, heh. [01:01:21] I also posted /notice messages in the old channel several times before the redirect was setup, but I don't know if that was really helpful or not. Contacting everyone pretty tricky. :/ [01:03:04] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:20:46] Oh right, I still can't get log in to Horizon. Hrm. [01:28:42] Yvette: login should be identical to wikitech [01:33:46] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3192230 (10Legoktm) >>! In T163284#3216516, @MusikAnimal wrote: > This discrepancy has been brought up many times over the years, and I feel like there must be a ticket somewhere... [01:43:03] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [01:44:49] mutante: Except for a mandatory two-factor code. [01:45:00] I remember where I left off. I guess I need to install http://www.nongnu.org/oath-toolkit/ [01:45:32] I wonder if that's installed on tools-bastion. [01:45:42] ah, well it was mandatory for me on wikitech too, so i didnt notice a chane [01:46:14] sounds like you dont want to use the google app [01:46:36] Not at all. [01:46:50] There's some discussion at https://phabricator.wikimedia.org/T161473 yeah. [01:47:03] I guess some people didn't notice because wikitech has been requiring it for so long for certain account types. [01:47:20] yes, that [01:47:39] so OATH != OAUTH.. [01:47:51] Yeah. [01:47:56] But https://wikitech.wikimedia.org/wiki/Help:Two-factor_authentication#Clients says it will work? [01:48:09] http://www.makeuseof.com/tag/5-best-alternatives-google-authenticator/ [01:48:13] ah, didnt know [01:48:17] I don't really want a mobile app at all. [01:48:20] If I can avoid it. [01:48:30] I'd rather not have 2FA altogether, but I doubt it's going away. [01:48:50] The original authenticator was open source and the open-source version still works fine. [01:48:53] There's also [OTP Authenticator](OTP Authenticator [https://f-droid.org/app/net.bierbaumer.otp_authenticator). [01:49:07] ah, i was just looking for better licenses [01:49:28] All these Android apps. Google runs the 2FA world, I guess. [01:50:43] because people want the second factor to be their phone usually [01:51:07] if you use the laptop it's not really a second factor, is it [01:51:14] I made that point on the task. [01:55:18] how would you prefer to use it? you just want 2fa disabled, or do you just want a better program to do it [01:55:36] I'd prefer no 2FA. [01:55:38] and what should be the second factor that isnt the same thing where the password is [01:55:44] But otherwise I'd like some non-phone way. [01:55:53] Lots of places have a hard token. [01:56:04] if pass factors are in the same place, might as well just have one [01:56:06] Like https://www.andrewpatrick.ca/wp-content/uploads/rsa_key.gif [01:56:13] you could use a yubikey [01:56:18] mutante: What about people who SSH with their phone? [01:56:21] but of course that would cost money [01:56:36] it would be separate hardware in your USB slot [01:56:40] and very secure [01:56:49] If you SSH with your phone, you're using the same device for second authentication? [01:57:05] Or SMS? [01:57:10] i would never put private keys on my phone [01:57:11] Or e-mail? [01:57:30] e-mail is also mandatory 2fa [01:57:32] for me [01:57:39] I mean why can't SMS or e-mail be two-factor for Horizon? [01:57:46] That's what my bank uses. [01:57:47] SMS could [01:57:54] Or a phone call. [01:58:04] SMS is less safe than the apps though [01:58:10] I want a Labs admin to personally call me every time I try to log in. [01:59:18] do you really not use 2fa on any other sites? it's kind of standard nowaays [01:59:38] My bank and credit cards sometimes require it. [01:59:43] But it's via SMS or e-mail. [01:59:50] I have a hard token for work VPN. [01:59:54] Otherwise, no. [02:00:05] I check e-mail on two desktop computers. [02:00:09] the SMS method has been broken [02:00:14] So have passwords. [02:00:20] Nothing is secure. [02:00:25] I just want to log in to Horizon. [02:00:32] I don't really care if it's secure. [02:00:43] well, we have to care if it's secure [02:01:12] mutante: Where do you keep your scratch codes? [02:01:20] GPG encrypted text file [02:01:33] well, for wikitech nowhere [02:01:40] because i know i can reset it anyways [02:01:46] Heh. [02:02:52] saying "nothing is secure" is not really super convincing. it could be used to argue against using https or ssh instead of telnet just as well [02:03:25] HTTPS and SSH don't require some stupid Google app. [02:03:40] I'll gladly offer up my SSH public key to log in. [02:03:51] then use the free software alternative that isnt google [02:04:05] oathtool? [02:04:20] FreeOTP ? [02:05:00] stupid mobile app * [02:06:05] mutante: Saying SMS has been broken when my bank (an actually important account) uses it, while Horizon doesn't (which is a completely unimportant account) is pretty silly, IMO. [02:06:07] btw, for yubikey, is there any plans for supporting? :P [02:06:11] (on wikitech or anywhere) [02:06:23] mutante: This isn't even like an AWS account that has some credit card info attached. [02:06:47] i'm trying to help, i didn't invent it or implement it [02:06:51] Sure. :-) [02:06:57] I'll look into oathtool someday. [02:07:01] There are Debian packages. [02:07:16] Might try to build it on tools-bastion. Otherwise, I can install it elsewhere. [02:19:21] sorry, i got disconnected because my phone was out of battery and it was my hotspot. now i just signed up for a "complimentary 1 hour pass" on some random xfinity wifi [02:20:35] oathtool - OATH Toolkit oathtool command line tool [02:21:13] yea, no. nevermind on that one [02:25:31] i'm actually asking #debian , will let you know if i find other alternatives [02:27:04] mutante: You think oathtool won't work? [02:27:07] It has Debian packages. [02:28:00] https://www.mediawiki.org/w/index.php?title=Help:Two-factor_authentication&oldid=2283920 hm. [02:32:13] PROBLEM - Free space - all mounts on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1417.diskspace._tmp.byte_percentfree (<100.00%) [02:43:41] Yvette: i will try it out myself [02:43:46] just got back online again [02:52:30] Yvette: https://5apps.com/gbraad/gauth ! [02:52:38] run it in your browser [02:52:41] and open source [02:53:53] Yvette: i found that from http://www.analyth.com/en/allarticles-en/gauth-and-oathtool-authenticator-2fa-for-gnulinux/ which has 2 examples, the GUI alternative in the browser above, and exanple how to use ethtool [02:57:30] i think that's the best answer for desktop for now.. the browser app https://github.com/gbraad/gauth [02:57:47] bd808 said he was running gauth, I think. [02:58:06] that would be that one then [02:58:44] http://gauth.apps.gbraad.nl/ shows an example to me in Firefox for alice@google.com [02:58:59] https://tools-static.wmflabs.org/bd808-test/gauth/ [02:59:00] Hm. [02:59:34] clicking add. it lets me add an account name and secret key [03:00:48] It stores all state in the browser's localstorage [03:01:28] I use it for testing oath stuff in striker.wmflabs.org [03:03:21] aah, so when i use the Firefox variant and install the "OpenWebApp" on Debian [03:03:47] it actually added an icon in gnome and all and i can start it like a real desktop app now as well [03:04:08] so there you go, Yvette [03:05:11] mutante: You installed gauth in Firefox? [03:05:15] As a plugin or something? [03:05:47] Yvette: yes, as an "app" , an OpenWebApp, from https://5apps.com/gbraad/gauth [03:05:58] and now it's a gnome desktop icon and in my menu [03:06:18] https://5apps.com/gbraad/gauth hmmmmm. [03:06:32] so it has Firefox app, Chrome app, ... [03:06:43] Yeah, this looks neat. [03:07:54] Hmmm. [03:08:24] Firefox downloaded a manifest.webapp file. [03:08:52] for me it started a dialog, asking me if i want to allow it to install [03:10:43] Firefox ESR 45.9.0 [03:11:26] https://www.mozilla.org/en-US/firefox/45.9.0/releasenotes/ [03:11:59] I'm not even sure what my goal is. [03:12:06] I need to scan some QR code, I think. [03:13:08] ok, there is a one-time setup step [03:13:13] Can this gauth app ingest this "Two-factor authentication secret key:"? [03:13:35] that involves adding an account and adding the key [03:14:05] and then later, when you login, you will just open it and look at the current number, and type it. since its time-based, the number will change like every 30 sec [03:14:24] Okay. [03:14:51] Is the webapp just a thin wrapper around http://gauth.apps.gbraad.nl/ ? [03:15:56] the part about adding the key is usually scanning a QR code, but we could also type it manually i suppose [03:16:53] it stores the config and secret in your browser's storage [03:17:26] So it's okay to give https://5apps.com/gbraad/gauth my secret key from https://wikitech.wikimedia.org/w/index.php?title=Special:Two-factor_authentication&returnto=Special%3APreferences ? [03:17:59] And I should write down the scratch codes, I guess. [03:18:25] yea [03:19:08] All right. [03:19:22] Thanks for the help. I'll try it out sometime and we'll see how it goes. [03:19:37] that should be a one time thing, you click "add account" , add the secret key that you get from wikitech [03:19:46] then you should have a new account called wikitech in it [03:19:57] and the numbers just keep changing and you use it at horizon login [03:20:31] If I change browsers, do I just re-enter the secret key from wikitech? [03:20:36] one big BUT.. though [03:20:48] Am I relying on localStorage? [03:20:54] if the app and the password itself are stored in the same desktop.. then it's all just one factor again [03:21:18] One factor is what I want. [03:21:43] yes, you are relying on localStorage [03:21:57] and eh.. that is not the idea of 2fa though [03:22:00] Right. So I'm trying to figure out what happens if localStorage goes away. [03:22:16] If I change browsers or delete the site settings. [03:22:21] In some cookie purge. [03:22:46] that would be like for us when we get a new phone, how it worked for me: [03:23:14] disable 2fa in settings, it asks you for code, you use backup code [03:23:24] enable it again, now add account in replacement device [03:23:30] Backup code is a scratch code? [03:23:33] yea [03:23:47] All right. [03:23:50] I think that'll work. [03:24:17] you should not store your password in that browser though [03:24:49] All the passwords go on Post-Its of course. [05:39:43] (03PS3) 10BryanDavis: Add toolinfo.json style data [labs/striker] - 10https://gerrit.wikimedia.org/r/353909 (https://phabricator.wikimedia.org/T149458) [05:39:46] (03PS1) 10BryanDavis: Add Code of Conduct links [labs/striker] - 10https://gerrit.wikimedia.org/r/357551 [05:59:13] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3321093 (10Samwilson) Once we're up and running at `xtools.wmflabs.org` will we keep `tools.wmflabs.org/xtools-dev` as the staging area? Or would it be better to have `xtools.wmflabs.org/dev`... [06:12:52] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Investigation: Add OAUTH for Xtools - https://phabricator.wikimedia.org/T165401#3322149 (10Samwilson) The first part of this, the authentication, is done; users can log in and out. [06:13:50] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3322150 (10Samwilson) Sorry, only just realised we've got `xtools-dev.wmflabs.org` for that. [06:21:45] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Restrict access to users' edit stats unless opted-in - https://phabricator.wikimedia.org/T165401#3322151 (10Samwilson) [06:32:35] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1422 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:42:19] revi: https://phabricator.wikimedia.org/T100373 [06:42:56] you can configure a yubikey to use TOTP, though [06:49:10] I once tried it but gave up [06:56:10] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1425 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:58:57] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:12:35] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [07:31:12] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:59] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [08:35:09] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Restrict access to users' edit stats unless opted-in - https://phabricator.wikimedia.org/T165401#3322431 (10Samwilson) Start of some docs: https://xtools.readthedocs.io/en/latest/opt-in.html And I've started integrating it into topedits etc. [08:37:11] 10Labs, 10Labs-Infrastructure: ssh fails for Labs instance with "debian-9.0-stretch (experimental)" image - https://phabricator.wikimedia.org/T167267#3322447 (10Magnus) [09:32:49] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:34:32] 10Tool-Labs-tools-Other, 10WMSE-Bug-Reporting-and-Translation-2017, 10Wikispeech-WMSE, 10User-LokalProfil, 10Wikispeech (Sprint 2017-01-25): New toollabs project not found in SAL - https://phabricator.wikimedia.org/T156127#3322576 (10Sebastian_Berlin-WMSE) [10:07:51] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [10:39:17] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:06:02] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10cloud-services-team (Kanban): Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3322895 (10Xelgen) @bd808 Frankly 2 weeks ago I've found workaround for djvutoxml - I copied binary and it's librar... [11:19:17] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:40:18] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:01:34] 10Labs, 10User-Addshore: Create new labs project for development & testing of the FileImport & FileExport extensions - https://phabricator.wikimedia.org/T161218#3323086 (10Tobi_WMDE_SW) [12:20:18] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:41:17] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:43:44] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3323568 (10chasemp) [13:44:09] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3323581 (10chasemp) [13:44:12] 10Labs, 10Operations, 10cloud-services-team (Kanban): Initial OpenStack Neutron PoC deployment in Labtest - https://phabricator.wikimedia.org/T153099#3323580 (10chasemp) [13:44:25] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3323568 (10chasemp) [13:46:36] 10Labs, 10Labs-Infrastructure: use a service role project for openstack components - https://phabricator.wikimedia.org/T167294#3323587 (10chasemp) [13:46:49] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3323600 (10chasemp) [13:46:51] 10Labs, 10Labs-Infrastructure: use a service role project for openstack components - https://phabricator.wikimedia.org/T167294#3323599 (10chasemp) [13:47:59] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3323568 (10chasemp) [13:48:01] 10Labs: Discontinue use of admin_token for keystone - https://phabricator.wikimedia.org/T167295#3323602 (10chasemp) [14:55:41] 10Labs, 10Labs-Infrastructure: Add more network nodes - https://phabricator.wikimedia.org/T41787#3324010 (10chasemp) 05Open>03Resolved a:03chasemp We did add a second network node to our single openstack deployment with a custom failover procedure, but more verbose versions of this will be tracked elsewh... [14:58:04] 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-Authentication-and-authorization, 10Reading-Infrastructure-Team-Backlog: Update OpenStackManager to use AuthManager - https://phabricator.wikimedia.org/T110461#3324065 (10Fjalapeno) [14:58:56] 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-Authentication-and-authorization, 10Reading-Infrastructure-Team-Backlog, 10MW-1.28-release (WMF-deploy-2016-06-14_(1.28.0-wmf.6)), 10Patch-For-Review: Update OpenStackManager to use AuthManager - https://phabricator.wikimedia.org/T110288#3324084 (10Fjal... [15:10:10] 10Labs, 10Operations, 10Patch-For-Review: Upgrade facter to version 2.4.6 - https://phabricator.wikimedia.org/T166203#3324156 (10Volans) 05Open>03Resolved a:03Volans Facter is upgraded in production on the whole fleet apart `cp3003.esams.wmnet,labstore[1001-1002].eqiad.wmnet` that will need to be reima... [15:13:36] (03CR) 10Andrew Bogott: [C: 032] Add Code of Conduct links [labs/striker] - 10https://gerrit.wikimedia.org/r/357551 (owner: 10BryanDavis) [15:15:18] (03Merged) 10jenkins-bot: Add Code of Conduct links [labs/striker] - 10https://gerrit.wikimedia.org/r/357551 (owner: 10BryanDavis) [15:16:16] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:18] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:57:47] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10cloud-services-team (Kanban): Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3324440 (10bd808) >>! In T166985#3322895, @Xelgen wrote: > As far as I understand in long term we plan to move from... [16:07:49] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3324460 (10Papaul) [16:08:24] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3324461 (10Papaul) [16:10:29] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3324462 (10Papaul) [16:17:17] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:22] !log admin-monitoring clearing out leaked instances — most are up and accessible and fine — this seems to be a timing issue of some sort [16:20:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [16:25:43] Hi, all. I am logged in bastion-02, ie. dev-tools.wmflabs.org. It is too slow. I feel It takes time a lot when I write or read on disk. [16:27:00] madhuvishy: ^ I'm in the ops meeting can you field this [16:28:11] chasemp, madhuvishy, I'll look [16:28:43] i can look too [16:29:20] * andrewbogott can't log at all so far [16:29:26] I assume we're talking about tools-dev not dev-tools? [16:29:59] * bd808 is looking [16:30:11] I think that whatever it was is done now [16:30:16] ryuch: better now? [16:30:28] (I didn't do anything but it seems fine to me) [16:30:40] maybe ... pstree took a stupid long time to return [16:30:48] but it does seem a bit more responsive [16:31:09] i don't see anything scary running [16:31:48] maybe a nasty puppet run? [16:31:51] hm, I may have spoken too soon, puppet sure is taking its time [16:32:39] "scp -r -f /data/project/icelab/EPJ_resub_revamp/" [16:32:48] ugh [16:33:01] chase kicked a big scp yesterday too [16:33:06] * andrewbogott looks up to see who 'icelab' is [16:34:23] this has to be nfs being saturated [16:34:38] * bd808 shakes fist at the past [16:34:39] that job is done actually, so it might not have been the culprit [16:35:22] is there a throttle arg you can pass to scp to reduce bandwidth? Or should I tell 'em to use rsync? [16:35:41] (update: I'm wrong, it's not done, just intermittently hoggish) [16:36:01] I walled a message to them to use rsync w/ bw limits yesterday [16:36:05] afaik scp is not able [16:36:36] chasemp: ah, same user? I'm emailing right now [16:36:41] yes [16:36:55] Did you say anything more detailed than 'use rsync with limits'? [16:37:47] not really, no [16:38:09] `scp -l RATE` -- "Limits the used bandwidth, specified in Kbit/s." [16:38:29] news to me! [16:38:40] but also for a bastion rsync soyou can resume seems essential [16:38:41] man scp for the win [16:39:06] whoah! [16:39:28] could be a user using some desktop too to copy things too :/ [16:39:31] we have crappy tools for moving data [16:39:38] bd808: amen [16:39:48] madhuvishy: Is there any reason I would ever use 'mviswanathan@wikimedia.org' vs 'madhuvishy@wikimedia.org'? I'm always confused when the two options show up in autocomplete. [16:40:06] andrewbogott: ah it's just an alias, use any of them :) [16:40:11] ok, I [16:40:24] I mean, ok! [16:40:40] bd808: want to follow up to my recent email with that -l RATE bit? [16:47:54] andrewbogott: not that much. I am on tools-bastion-02. [16:56:47] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#3324705 (10yuvipanda) a:05yuvipanda>03None [16:56:50] 10PAWS, 10Pywikibot-core: How to change the default uploading time in PAWS - https://phabricator.wikimedia.org/T147832#3324707 (10yuvipanda) a:05yuvipanda>03None [17:01:50] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3324762 (10Papaul) [17:01:54] 10Labs, 10Labs-Infrastructure, 10Operations, 10netops, 10ops-codfw: codfw: labtestpuppetmaster2001 switch port configuration - https://phabricator.wikimedia.org/T167321#3324750 (10Papaul) [17:05:40] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3324790 (10Papaul) [17:05:44] 10Labs, 10Labs-Infrastructure, 10Operations, 10netops, 10ops-codfw: codfw:labtestnet2002 switch port configuration - https://phabricator.wikimedia.org/T167322#3324778 (10Papaul) [17:06:50] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3324804 (10yuvipanda) @Andrew right. However, it's already the case tho - PAWS right now is still mostly reliant on me, for mostly r... [17:10:56] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Fix or delete tools-worker-1028 and 29 - https://phabricator.wikimedia.org/T167324#3324813 (10yuvipanda) [17:12:34] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3324856 (10Papaul) [17:12:38] 10Labs, 10Labs-Infrastructure, 10Operations, 10netops, 10ops-codfw: codfw: labtestneutron2002 sswitch port configuration - https://phabricator.wikimedia.org/T167326#3324843 (10Papaul) [17:13:16] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:16:51] 66.67% is critical? [17:17:53] there is still a huge replag on commonswiki_p; sigh! [17:21:01] that will be hopefully fixed when the new dbstores for labs are setup fully Steinsplitter [17:21:32] Zppix: How long it will take? is there any workaround? [17:22:18] chasemp: told me of a workaround i cant remember it though Steinsplitter [17:23:53] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3324942 (10yuvipanda) @madhuvishy if this happens, I'll also need to transfer the entire contents of the paws tool dir on tools NFS... [17:26:35] Steinsplitter: it looks like there is some table alter maintenance happening for the s4 shard in prod -- T166206 -- https://tools.wmflabs.org/sal/production?q=s4 [17:26:35] T166206: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206 [17:27:14] ah, :) [17:31:17] there is another set of labsdb servers that may be less affected. Now if I could just remember how to access them... [17:32:23] you have to spcify by server address i think in the query or whatever [17:32:47] madhuvishy: do you remember the service names that folks need to use to read from the new labsdb servers? [17:33:12] bd808: uhhh one sec [17:35:04] labsdb-web.eqiad.wmnet [17:35:12] madhuvishy: i just found it damn you xD [17:36:17] tangentially related, there are some big prod schema changes planned that people will need to update their SQL queries for -- T166732 -- and some discussion of possible ways to bridge this in the short term -- T166798 [17:36:18] T166798: Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas - https://phabricator.wikimedia.org/T166798 [17:36:18] T166732: Refactor comment storage in the database and abstract access in MediaWiki - https://phabricator.wikimedia.org/T166732 [17:36:22] thanks madhuvishy [17:37:13] it doesn't have all the shards yet afaik but should have s4 [17:37:35] Steinsplitter: ^ labsdb-web.eqiad.wmnet may get you more current data [17:37:54] bd808: Thanks! will use that host then. :) [17:38:20] that replica is using a different setup that hopefully does not lag as badly during schema updates [17:38:57] sometime in the next 3 months we hope to make that server cluster the default [17:39:09] but it is still filling up with data for some shards [17:39:10] :-) [17:39:47] T153743 [17:39:48] T153743: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743 [17:40:18] how will it catch up? is it doing it 24/7 at an high rate? [17:46:11] https://phabricator.wikimedia.org/search/query/Xaz8r3IxKEqq/#R icelab yet again :( [17:47:46] Zppix: yes, basically [17:48:16] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:48:52] zhuyifei1999_: hmmm I wonder if opening T136197 prompted someone to try and grab it all [17:48:52] T136197: icelab is using 245G in Tools - https://phabricator.wikimedia.org/T136197 [17:49:16] that task has been around for a year? [17:50:49] madhuvishy: welcome to our backlog! I jsut saw "Jun 2" and ignored the year :/ [17:51:00] I would think scfc grabbed some of them in https://phabricator.wikimedia.org/T157855#3018984 [17:51:13] bd808: me too lol [18:00:18] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3325081 (10madhuvishy) @yuvipanda Yeah that's fine by me. PAWS currently uses 51GB in the tools share, and we've talked about moving... [18:02:19] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3325099 (10yuvipanda) @madhuvishy +1, I'd love it to be a separate share! [18:06:27] !log git gerrit-test3 saving backup of gerrit 2.14 and downgrading to 2.13.8 to try and reproduce T152640 [18:06:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [18:06:30] T152640: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640 [18:39:18] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:56:52] andrewbogott: the load on the bastion is still really high [18:57:01] i'm considering killing the scp [18:59:13] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#3326586 (10Soni) Just wanted to point out that 4 months later, the queries are still running. [19:02:51] 10PAWS, 10Pywikibot-core: How to change the default uploading time in PAWS - https://phabricator.wikimedia.org/T147832#2704902 (10Mpaa) Try: ``` import pywikibot.config2 as config config.put_throttle = 60 ``` [19:05:51] !log tools Killed scp job run by user torin8 on tools-bastion-02 [19:05:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:19:15] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:45:14] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330053 (10MusikAnimal) [19:45:44] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330066 (10MusikAnimal) p:05Triage>03Low [19:58:34] madhuvishy: don't feel bad about killing, I already said we would in my email :) [20:00:51] its not like they cant just resume it with better limits [20:01:36] 10Quarry: Explain command forces Quarry to keep running endlessly - https://phabricator.wikimedia.org/T155808#3330162 (10Nemo_bis) I think what the interface says isn't necessarily true. [20:02:36] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330166 (10Framawiki) [20:03:39] 10Labs, 10PAWS: Move PAWS nfs onto its own share - https://phabricator.wikimedia.org/T160113#3330181 (10yuvipanda) [20:03:43] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3330180 (10yuvipanda) [20:06:00] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3330192 (10RobH) [20:06:04] 10Labs, 10Labs-Infrastructure, 10Operations, 10netops, 10ops-codfw: codfw:labtestnet2002 switch port configuration - https://phabricator.wikimedia.org/T167322#3330189 (10RobH) 05Open>03Resolved a:03RobH Done! [20:06:21] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3330193 (10RobH) [20:06:31] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3319490 (10RobH) [21:09:02] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330397 (10MusikAnimal) [21:11:00] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330053 (10MusikAnimal) To put in some !votes, I like "XTools" the most, but I'm not married to it, and I believe @Matthewrbowker likes "xTools" and @Samwilson prefers "Xtools". We're on our way to a c... [21:18:05] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330053 (10DannyH) I thought we settled this at the hackathon, with me as tiebreaker. I vote for XTools. [21:22:59] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3330482 (10chasemp) [21:23:01] 10Labs: Undo trunking to virts for a sane flat networking model - https://phabricator.wikimedia.org/T167356#3330470 (10chasemp) [21:32:29] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3330515 (10chasemp) [21:32:34] 10Labs: Determine need and replacement for dmz_cidr configuration in nova-network - https://phabricator.wikimedia.org/T167357#3330503 (10chasemp) [21:35:19] 10Labs: Nova-network to Neutron migration - https://phabricator.wikimedia.org/T167293#3330526 (10chasemp) [21:35:23] 10Labs, 10Operations, 10netops: Consider renumbering Labs to separate address spaces - https://phabricator.wikimedia.org/T122406#3330525 (10chasemp) [22:02:23] bd808, we all should read regex fluently because the world is super freaking messy. :) [22:02:41] but props for breaking it down [22:03:05] * bd808 has a minor in DFA and NDFA notations [22:03:56] halfak: this bash quip is related -- https://tools.wmflabs.org/bash/quip/AVWoDg8ZgCrwkbTdmcjL [22:04:21] lol right [22:04:39] Dan was asking about BeautifulSoup [22:04:43] Just that expecting to not be in garbage ocean is apparently asking to much. Evidence: look around :( [22:05:16] Even the most well meaning stuff turns into regex garbage given enough time. Pass me a pitch fork! [22:07:39] I have killed a lot of PERL and PHP regex use in my 4 years here [22:09:06] That's some good work. [22:09:18] But it's like cleaning up garbage. There will always be more. [22:09:28] Gotta stay diligent [22:09:43] Unrelated, there's a big ranty thread here about replication issues in labs DB: https://www.mediawiki.org/wiki/Topic:Trtvgc5uuxat7ktj [22:10:47] halfak: thanks for pointing that out. I'll give them some response [22:11:33] Great. Thanks :) [22:33:31] joined [22:33:55] @halfak [22:33:55] Hey folks. I'm helping QZanden get set up on labs/cloud with an shell username [22:34:05] What's the best reference for newcomers these days? [22:34:11] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330672 (10Samwilson) I think it should be "eggs' tools", like tools for eggs, because //who thinks of the eggs//?! In more general terms, I'm also happy with XTools. [22:34:21] halfak: https://wikitech.wikimedia.org/wiki/Help:Getting_Started [22:34:26] QZanden: ^ [22:34:36] also welcome :) [22:34:38] thanks madhuvishy :) [22:34:40] \o/ [22:34:44] ONE OF US! [22:35:48] QZanden, https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Creating_an_account_2 [22:35:52] Start there. [22:36:01] We are a VPS General project [22:36:24] the tool users workflow is probably the nicest if you don't have a wikitech account yet [22:36:55] Oh. That's cool too. We won't be working in tools with Snuggle, but if you can access tools you can work with Snuggle. [22:36:58] you can just stop after creating the account with the wizard at https://toolsadmin.wikimedia.org/register/ and come back to wikitech for other things [22:37:04] Snuggle should eventually live in tools [22:37:16] BUT no time to make that happen yet [22:38:22] But at wikitech I have already an account. That second link you gave, is one to create at wikitecht [22:38:52] if you have a wikitech account already then you can probably skip to https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Uploading_an_SSH_key [22:39:35] if you have an ssh key listed in wikitech already then you can probably jsut poke halfak to add you to the VPS project that you will be working in [22:40:29] bd808, can I add people in horizon or is that still wikitech only? [22:40:29] I'm not very familiar with using the command-section of a computer, is it really necessary to do that? [22:40:59] halfak: project membership is still only in wikitech. [22:41:14] QZanden, yeah. We're going to need to do a lot of that. [22:41:25] thanks bd808 [22:41:27] QZanden: if you are going to work on halfak's VM you will need to learn some ssh [22:42:10] we are around to help if you get stuck though! [22:42:23] https://wikitech.wikimedia.org/wiki/Help:Access as some details on using SSH [22:42:32] Thanks! [22:42:43] there are probably much better tutorials you can find with a google search though [22:43:05] QZanden: what operating system do you use? I can look for a good tutorial for you [22:43:31] windows 10, 64x [22:43:50] dell computer [22:45:00] this looks like a pretty good overview of PuTTY (windows SSH client) -- https://support.suso.com/supki/SSH_Tutorial_for_Windows [22:50:14] QZanden, I successfully added you to the project. [22:50:44] I've got to run now, but next time we meet up, I'll help you connect to the instance and show you how to manage a Snuggle instance. [22:51:14] downloaded PuTTy, what hostname should I pick? something with wikimedia? [22:51:18] Sorry to run away when you're learning something new (ssh), but just try to learn what you can in the meantime. [22:51:42] QZanden, good Q. We're both going to need to learn something about putty to do the next step. [22:51:52] I have no idea how configuration works in putty :( [22:52:16] @halfak, never mind. Do I still need to learn SSH when you've added me to the project? [22:52:23] Yes. [22:52:36] I'll do my best! [22:53:04] Make sure to get your ssh key in wikitech and that'll be good progress. :) [22:53:06] o/ [22:55:32] QZanden: this may help -- https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [22:56:15] the screen shots are very out of date :/ the pmtpa servers have been gone for 3 years [22:56:29] bastion.wmflabs.org should work for you [22:58:23] hi [22:58:33] QZanden: if you keep track of what works and what doesn't that would be awesome. I'll totally send you stickers in exchange for updates to the wiki docs :) [22:58:41] hey matanya. how are things? [22:59:00] i don't know why, but some of my scripts tend to die with : [22:59:04] pywikibot.data.api.APIError: readonly: The database has been automatically locked while the slave database servers catch up to the master [readonlyreason:Waiting for 5 lagged database(s); help:See https://he.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.] [22:59:11] over and over [22:59:29] do we have lagging issues in prod ? [22:59:50] and of course the error python gets : [22:59:51] CRITICAL: Closing network session. [23:00:52] so i am not sure if this is the right venue bd808 [23:01:09] but other than that, all things are cool :) [23:02:04] matanya: there is some maintenance on s4 (commons) that may be causing replag in prod [23:02:36] I think there is a way to tell pywikibot how much lag to tolerate... [23:03:12] -maxlag apparently -- https://www.mediawiki.org/wiki/Manual:Pywikibot/Global_Options [23:03:12] Is there anyone here with some experience using PuTTy with wikimedia/ [23:03:28] or wikitech? [23:03:29] QZanden: why would you want to do that ? [23:03:39] thanks bd808 i will look into it [23:03:59] matanya: because they are a new windows user of Cloud Services. no OS shaming please [23:04:02] I'm trying with halfak to set up Snuggle at wikidata. [23:04:29] QZanden: I have used PuTTY very very long ago [23:04:30] btw, bd808 your previous hint about pywikibot enabled: https://commons.wikimedia.org/wiki/Category:Content_media_by_years_-_Supported_by_Wikimedia_Israel_-_2017 [23:04:39] QZanden: I haven't used it, but happy to help debug [23:04:43] bd808: no OS shaming, was really wondering [23:04:49] and welcome QZanden [23:04:59] Thanks, matanya! [23:05:13] matanya: :) I get touchy about anti-Windows stuff [23:05:25] @bd808, what hostname shoul I use to get connected with wikitech? [23:05:45] QZanden: have you created an ssh key already? [23:05:48] bd808: i am on CoC, i can't even be anti nowadays, I wonder who got me there ... [23:06:48] matanya: that one wasn't my fault I swear [23:06:51] no, first tried to get putty working [23:07:11] bd808: which one is your fault ? [23:07:28] the tools committee [23:07:34] fair [23:07:54] QZanden: once you create your ssh key, i might be able to walk you through [23:08:01] QZanden: ah, I think you first need puttygen in order to make an ssh key [23:08:09] indeed [23:08:17] QZanden: this may help you -- https://wikitech.wikimedia.org/wiki/Help:Putty -- the ssh key is a necessary early step [23:08:55] * bd808 wonders briefly why we have multiple PuTTY pages and then remembers this is wikitech [23:08:58] i think the confusing this is puttygen.exe is a separate downloadable [23:09:14] s/this is/thing is [23:09:31] it is available here - https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html [23:10:14] now generating an shh-key. Taking some time... [23:10:27] QZanden: cool! [23:11:04] QZanden: just for future ref, mobaxterm is more user friendly for new comers [23:12:22] matanya: oh neat. I haven't seen that one before. It even has mosh support [23:12:52] TIL mobaxterm [23:12:56] yes [23:13:00] @matanya, is it worth now switching to mobaxterm instead of keeping putty? [23:13:34] if you want to get up and running rather than understanding the protocol, yes [23:14:13] Well, maybe it is a good thing to learn something from this Pu [23:14:19] TTy stuff... [23:14:34] accidently hit [23:15:42] bd808: more ssh clients you never heard of :https://en.wikipedia.org/wiki/Comparison_of_SSH_clients and https://www.htpcbeginner.com/best-ssh-clients-windows-putty-alternatives/2/ [23:28:40] Windows SSH clients that aren't PuTTY?? [23:29:17] command-line ssh? [23:29:48] install Linux subsystem and just use vanilla ssh? [23:30:48] harej, what's wrong with putty? [23:31:05] MaxSem: Nothing; I'm astonished that non-putty clients exist :P [23:36:32] MaxSem: for that you need windows 10 [23:57:07] considering you can't buy any other version anymore...