[10:02:57] (03CR) 10Jean-Frédéric: [C: 032] Stricter parsing for identifying Phabricator tasks [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/414516 (owner: 10Lokal Profil) [10:04:09] (03Merged) 10jenkins-bot: Stricter parsing for identifying Phabricator tasks [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/414516 (owner: 10Lokal Profil) [10:05:29] (03CR) 10jenkins-bot: Stricter parsing for identifying Phabricator tasks [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/414516 (owner: 10Lokal Profil) [10:23:14] !log tools disable puppet in tools-proxy-01 for apt pinning tests [10:23:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:35:47] !log tools enable puppet in tools-proxy-01 [10:35:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:43:19] anyone knows how I can run npm on tools wmflabs? [15:43:30] i don't have privileges [16:02:50] Ester: have you tried using NVM? [16:05:06] bstorm_: no I have never used it [16:05:30] That might give more control, but are you trying to do global installs? [16:05:33] npm -g? [16:05:41] Since npm will generally work on local stuff [16:05:54] no i'm not [16:06:06] hmm. In that case, let me check something [16:06:25] I think you should be able to run npm on something in a project to install a package.json [16:08:19] If you don't need global installation (-g) I don't think you'd need privs other than login and a project. If you are trying to get functionality that you'd see in a global install, there is https://coderwall.com/p/6aumug/howto-npm-global-install-without-root-privileges [16:08:58] nvm is mostly just if you need to mess with node versions [16:09:45] Is it a particular project you need it run in? [16:10:01] I might not be understanding [16:10:24] bstorm_: thanks a lot, trying to figure things out and ?ll write back soon [16:10:31] bstorm_: thanks a lot, trying to figure things out and I'll write back soon [16:11:05] Np, let us know [16:15:26] Ester: one issue is npm/node in bastion host are ancient [16:16:16] Your best bet is to use a k8s shell with the nodejs image to install (and run) node.js stuff [16:17:14] That means `webservice --backend=kubernettes nodejs shell` [16:17:50] oh i see. a doubt: do i need to run npm start? [16:17:52] i'm confused [16:19:33] Npm start is a command to run whatever was specified as 'start' in the package.json file [16:20:14] that seems obvious now :) :) [16:20:21] Maybe I can help you better if we start from what it is you're trying to do [16:20:30] ok [16:20:33] thanks [16:21:21] I have a github repo with some code that is running here https://tools.wmflabs.org/etytree/etymology/ [16:21:31] the repo is here https://github.com/esterpantaleo/etymology/ [16:21:38] I recognize our node.js support and docs are not great at the moment [16:21:44] I'm trying to install a new branch [16:21:48] that uses webpack [16:21:52] https://github.com/esterpantaleo/etymology/tree/webpack-branch [16:22:01] were the tool is now [16:23:05] trying to install [16:23:08] right now i get Error: Cannot find module 'sync-exec' [16:23:15] but it's still running [16:23:57] lots of errors now [16:29:51] Ester: I'm on mobile, sorry if I'm taking a while to respond [16:29:55] Let me check [16:32:15] Unfortunately I have to go now [16:32:25] how can we keep in touch? here later? [16:32:29] or by email? [16:33:12] I'll be here, you can email the list if that is better for you [16:33:39] ok thanks [17:50:31] Hey cloud team, fr.wikipedia.beta.wmflabs.org uses an invalid security certificate. [17:51:22] The certificate is only valid for the following names: [17:52:25] Should I file a task? bd808 [17:52:34] Keegan: a task would be perfect. [17:52:45] will take a look at what's going on [17:52:54] Roger dodger [17:54:18] that's a fancy letsencrypt cert I think... [17:54:46] Probably the work of Krenair originally [17:55:43] bd808: yeah .beta.wmflabs.org - is that a thing we manage? I can't find in our proxies at lease [17:55:45] least* [17:55:56] some deployment-prep specific setup? [17:56:09] no, the deployment-prep project has a public IP and varnish etc [17:56:31] right [17:56:31] so technically it's a releng thing [17:56:52] https://github.com/wikimedia/puppet/blob/production/hieradata/labs/deployment-prep/host/deployment-cache-text04.yaml [17:57:24] yeah looks like it, Keegan you might wanna poke -releng, or if you have a task, I can poke them :) [17:57:51] I was just coming here to ask what tags I should use for the task. [17:58:05] https://phabricator.wikimedia.org/T188288 [17:58:30] I would guess this is a new wiki that they added and it needs to be added to that list madhuvishy found to get added to the cert [17:58:43] yup [18:00:46] madhuvishy: Should I poke releng or are you? [18:01:25] Keegan: I did [18:01:34] Cool, thanks y'all [19:17:58] !log tools tools-clushmaster-01:~$ clush -w @all "sudo puppet agent --test" [19:18:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:21:26] i just saw backlog here about fr.beta.wmflabs.org and a missing cert [19:21:46] the fix was likely just a simple puppet change to add the name to a list (greg uploaded it, i merged that) [19:22:19] but .. it cant be applied becase puppet on deployment-cache-text04 is broken due to other unrelated reasons .. which are kafka/webrequest related [19:22:47] and i pinged analytics ops about it [19:23:28] once puppet runs there again .. it _should_ also fix the cert issue above [19:27:36] hi, puppet seems to be down [19:27:41] it is failing with [19:27:41] Feb 26 19:05:32 gerrit-mysql puppet-agent[31835]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Reading data from Hosts/gerrit-mysql failed: Errno::ENETUNREACH: Network is unreachable - connect(2) for "wikitech.wikimedia.org" port 443 (https://wikitech.wikimedia.org:443) at /etc/puppet/manifests/realm.pp:22:14 on node gerrit-mysql.git.eqiad.wmflabs [19:28:28] andrewbogott i guess your working on wikitech? [19:28:32] paladox: if wikitech has an issue then puppet will have issues and wikitech was the middle of a transition [19:28:37] I'm not sure on exact current state [19:28:40] ah i see [19:28:42] thanks [19:29:14] it should be working fine now [19:33:04] thanks. [19:48:32] andrewbogott it seems it's still broken [19:48:33] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Reading data from Hosts/gerrit-mysql failed: NoMethodError: undefined method `[]' for nil:NilClass at /etc/puppet/manifests/realm.pp:22:14 on node gerrit-mysql.git.eqiad.wmflabs [19:51:46] works now, after restarting the host. [19:55:30] paladox: fyi another try will happen later today so it'll be flappy [19:55:36] try at moving wikitech DB tha tis [19:55:57] that sounds like the error would happen when it cant talk to Hiera "(Reading data from Hosts/" due to the wikitech Hiera backend [19:56:20] ok thanks for letting me know :). [19:56:23] could possibly move Hiera stuff to repo backend instead [19:56:47] and then not rely on the webserver [20:00:43] mutante: that's the long term plan, to dicontinue wikitech as a hiera backend [20:11:53] chasemp: ah!:) +1 i guess. while it was convenient it also added to confusion to have multiple places for it [21:09:43] andrewbogott, ah, wikitech db is locked till you complete swat this evening? [21:10:16] subbu: hopefully not that long, but for a while yet [21:10:24] hi friends! is there a central place for licensing information for wikimedia ops stuff? we're considering using e.g. https://github.com/wikimedia/puppet-zookeeper but can't find any accompanying license information. [21:11:06] ok. i tried editing the deployments page, and i got the msg that it is locked for migration .. thought i would check. can you ping me once that completes? i wanted to add 3 config patches for the evening's swat deploy as well. [21:11:11] (by corporate policy, we need a license, and i think a lot of companies probably fall under that) [21:15:03] subbu: I will try to remember — ask me again in ~90 minutes if I haven't responded by then [21:15:21] will do. [21:16:32] impl: the closest I know of is the requirement to run on cloud being that it is an OSI approved license https://opensource.org/licenses [21:17:21] hm, so this project is effectively unlicensed then? [21:18:40] !log tools Deleted tools-static-10 and tools-static-11 now that they are replaced with the much smaller 12 and 13 https://phabricator.wikimedia.org/T182604 [21:18:42] bstorm_: Failed to log message to wiki. Somebody should check the error logs. [21:18:55] Hrm. [21:19:07] impl: that licensed (or unlicensed) status of our puppet code is a long-running problem. I think that chasemp did the last investigation of same but if you ask him about it he'll probably just shake is head and take the rest of the day off. [21:19:15] bstorm_: wikitech maintenance in progress means that it will probably fail [21:19:21] That said — that particular module is small enough that you might be able to round up all the contributors and get them to sign off on something [21:19:24] Ah ok :) [21:19:33] Missed that [21:19:51] 90% of this downtime is just me copying giant files around :( [21:19:59] Fun! :) [21:20:02] andrewbogott: ahh, that's a shame -- it's nice code! [21:20:21] impl: it's indeed a long running issue [21:20:38] impl: I encourage you to rescue it if you can. The main puppet module is mostly hopeless due to contributors that are now hard to reach… but that one might be possible. [21:20:47] https://gerrit.wikimedia.org/r/c/183862/ [21:21:08] Amusingly enough, SAL is showing it on my task :) [21:22:45] bstorm_ it logs to phabricator too. [21:22:58] I noticed [21:22:58] which is not connected to wikitech (except if users use ldap for login) [21:23:02] yeah, hm, since this is standalone and only has a few people committing to it maybe we can get something pushed through [21:33:57] impl: I would write a proposed patch (like chase's) and get all the contributors to +1 [21:34:27] and ask here or in -operations if you need help running any of them down [21:35:45] ok, sounds good. thanks, I'll give that a try [21:39:52] impl: if they are indeed all WMF or ex-WMF employees it should be very doable as we all have our code licensed appropriately as part of employment I believe [21:44:09] there's a few sketchy ones (i'm not sure who's who exactly, but mostly people are still active): https://phabricator.wikimedia.org/p/ctrochalakis/ https://phabricator.wikimedia.org/p/Gage/ and a Mathias Klette [21:45:37] looks like Gage is ex-WMF [21:47:53] yes and he didn't contribute post-employment to my knowledge impl [21:49:36] https://phabricator.wikimedia.org/T67270 is the task for puppet licensing [21:51:34] and we should totally work on that :( [21:51:45] I will send an email to ctrochalakis and Mathias Klette to hopefully get their sign-off under Apache-2.0, and i think the rest are WMF. [21:52:16] impl: awesome, thanks for putting in the effort there [21:52:30] ctrochalakis will not be a problem [21:52:35] he's a friend of mine [21:52:54] the other contribution is trivial, and probably not copyrightable, but it doesn't hurt to ask [21:53:16] impl: can you Cc me? faidon@ [21:53:24] awesome, thank you. will do [21:53:51] so that it stays on our record, and I can merge the license addition [21:54:05] thank you so much and sorry to put you through this [21:54:59] no worries! i'm glad to help! [22:13:47] bd808: So, everytime someone has filed a security bug about some tool doing something silly, I've been adding you as a subscriber. Is that actually useful, or would you prefer I didn't do that? [22:15:04] bawolff: I'm not sure honestly. I guess its good to know, but unless its a really horrible thing I'm probably not going to have time/energy to be involved. [22:15:57] ok. If at some point it becomes annoying just let me know and I'll stop :) [22:16:02] bawolff: it seems like something that might be in scope for the toolforge standards committee? [22:16:14] (i am become scope creep, destroyer of worlds) [22:16:18] harej: that's a great idea :) [22:16:49] It seems consistent with a general notion of "standards" [22:17:04] and everyone in the toolforge standards committee is under NDA [22:17:10] bawolff: are you referring to https://phabricator.wikimedia.org/T188323? [22:17:13] Also, what is our opinion of gadgets having wmflabs.org dependencies? Is it okay because gadgets are optional? What if they're opt-out gadgets? [22:17:18] Well from a security person standpoint, the ideal situation is bug comes in, I see its toolforge related, and then I have a person I can cc on the bug to hand it off to [22:17:20] so not a "leak" problem [22:17:34] and then I check back in like a couple of weeks to make sure it got "handled" in some sense [22:18:13] bawolff: in principle you should be copying the tool maintainer/developer who is responsible for their code being orderly. however if they're MIA then it would fall on the standards cmte by default [22:18:59] I would be in favor of putting follow up/tracking down maintainers in scope for the committee [22:19:18] what do y'all do these days anyway? ;) [22:19:29] zhuyifei1999_: That is what triggered this particular question, but I'm more wondering about the general case then in any specific bug [22:19:35] k [22:19:43] We recently put a tool under receivership. We also changed our name from Tool Labs to Toolforge. [22:20:42] cloudforge :P [22:21:09] toolicorn (tools + unicorn) [22:21:28] I'm just making up silly names now to troll [22:22:00] * bd808 sends bawolff back to triaging Phab instead of trolling [22:22:05] I'm sure bd808 can give you a detailed explanation of why we did or did not go with a particular name, from his hours of conversations with branding people and lawyers [22:22:14] bd808: 'kay 'key ;) [22:22:28] harej: I know, that's what makes it such a good trolling target [22:22:46] harej: you'd like to think there was that much planning involved wouldn't you. ;) [22:24:19] * harej made peace with the whole cloud thing upon realizing it was the least bad option [22:24:31] as did we all friend [22:24:46] So but to summarize, What I got out of this conversation: in the event that there's a security bug related to a toolforge project - I should make sure that the author (if i know who that is) and folks from the toolforge standards comittee are cc'd on the bug. If the bug lingers for a long time (for some unspecified definition of long), I should follow up with toolforge std comittee folks [22:25:01] is that correct? [22:25:15] bawolff: if it's really seroius and you feel like something needs to be disabled or cutoff now ping wmcs folks :) [22:25:31] Yep yep [22:25:41] This is for the "tool is doing something stupid" case [22:25:55] not the, "tool is causing a site emergency somehow" case [22:26:44] understood! [22:26:47] bawolff: that sounds like a good plan to me. Should we write it up somewhere to make it "official" [22:26:52] ? [22:27:18] Umm. Perhaps. I'm not sure we actually have a place [22:27:32] I guess we could write it up as a subpage of Wikimedia Security Team on mw.org [22:28:12] there are a few places I can think of to reference such a policy actually, probably a great idea to do so [22:28:16] harej: would/could you add something to the Toolforge Standards Committee page about it? [22:28:32] * bd808 manages and delegates [22:30:07] isn't that the good quality of a boss, to know how to/whom to delegate? [22:30:24] Hauskatze: that's what I keep telling myself [22:30:47] bd808: in that case you're the head of WMCS iirc so you can do that :) [22:31:59] also, lol. I guess I never read the Wikimedia security team pages. Right at the front of the page there is https://www.mediawiki.org/wiki/Wikimedia_Security_Team/Volunteers talking about how me and reedy are volunteers [22:32:08] only a tad outdated [22:32:21] * bd808 promotes bawolff and Reedy [22:33:05] bawolff: omg. your boss has a redlink for a user page [22:33:17] that's worth taunting him about [22:33:45] dude is slackin' [22:35:07] never heard of Mr Bennett [22:36:01] He's a relatively new hire. I don't know that I've seen him announce himself on wikitech-l yet. [22:36:22] I think victoria sent an announcement [22:36:27] or was that only to wmfall [22:36:37] He's had a pretty busy first month since he joined pretty much the day that we kicked off annual planning [22:36:50] there was an email around christmas with a few lines but not much else I think [22:37:27] * bd808 is reminded that he was supposed to write a "welcome our new 🦄s" blog post [22:37:43] bawolff: https://phabricator.wikimedia.org/maniphest/task/edit/form/2/ doesn't let you change the tags. What is better: that a filed task not have any additional/useful tags (beyond Security) until later, or that the task has the correct tags but does not have the security access policy applied until later? [22:39:54] You should file the bug first, and add tags later [22:40:18] That is file, with form 2 so its security protected from the get go [22:40:56] Assuming its sensitive. If its not sensitive it doesn't really matter what you do [22:40:56] bd808 what's "welcome our new 🦄s"? Is that toolforge? [22:41:27] bawolff: i imagine security issues are inherently sensitive? [22:41:38] There's a gradient [22:42:09] but if you're writing a guide, you should tell people to just file it as a security bug. [22:43:21] Some security issues are inherently not really sensitive. E.g. something like, change the password hash algorithm to use more rounds. Its security related, but changing the safety margin from something that's safe, to something that's more safe, doesn't really need to be secret [22:44:46] paladox: new folks on the Cloud Services extended team [22:44:56] ah i see. thanks. [22:44:57] But "this tool is using unsalted MD5 hashes" is more urgent I imagine [22:47:39] bawolff: what happened to darian? [22:48:16] Hauskatze i think he left. [22:48:29] His account looks disabled on phab. [22:48:34] I know that :) Do we know the reason? [22:48:48] bawolff: But "this tool is using unsalted MD5 hashes" is more urgent I imagine [22:48:53] Hauskatze: Yes, he left around mid-hanuary [22:48:58] Hauskatze i think wmf does, but i doint think they will disclose that in public [22:49:19] nothing like going to the source then [22:49:27] harej: otoh I'm hoping that tools don't collect user passwords at all [22:50:48] Hauskatze: Asking a buddy, so to speak [22:51:14] hm? [22:55:09] !log stashbot Test [22:55:09] bd808: Unknown project "stashbot" [22:55:09] bd808: Did you mean to say "tools.stashbot" instead? [22:55:15] !log tools.stashbot Test [22:55:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [22:55:26] * bd808 hig fives stashbot [23:05:51] andrewbogott i think hosts in labs may get this error [23:05:53] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Reading data from Hosts/gerrit-mysql failed: NoMethodError: undefined method `[]' for nil:NilClass at /etc/puppet/manifests/realm.pp:22:14 on node gerrit-mysql.git.eqiad.wmflabs [23:06:00] see -releng [23:06:21] and i am getting warnnings from icinga2 for ores* and git* [23:06:42] i think the fix is to reboot. [23:06:52] wikitech is down [23:07:16] oh [23:07:27] (Cannot access the database: Cannot access the database: Can't connect to MySQL server on '208.80.154.136' (111) (208.80.154.136)) [23:10:10] 154... that won't be a labs instance [23:10:22] oh it's silver, wikitech [23:10:40] I believe wikitech is looking for mysql in the wrong place atm due to scap shenanigans [23:10:40] reading up, makes sense [23:10:50] andrew is trying to unwind the issue [23:12:40] kinda surprised I didn't learn that IP address seeing as I used to interact with it so frequently [23:13:12] not sure what they did to silver today [23:13:20] Hauskatze db migration [23:13:23] they changed wikitech from .. [23:13:28] ok [23:13:32] from silver to another host [23:13:53] well, I'm also migrating from my desk to my bed [23:13:57] see you [23:36:11] Is there another query killer killing after 90 seconds? [23:46:20] * impl scratches head at jenkins-bot, continues to cause havoc in gerrit