[00:16:10] if I wanted to access the DB replicas from my own computer, what would I need to do to map "enwiki.labsdb" to a real-world address? [00:16:44] harej: I think there is something about that on wikitech... [00:17:13] last time i asked about this i was told that rfemote access is not possible [00:17:41] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connecting_with... [00:17:56] (i wanted to use mysql workbench) [00:18:12] Danny_B: that's what is documented. I haven't tried myself [00:18:24] basically you need to setup ssh tunnels [00:18:44] and mysql workbench I guess has native support for that [00:19:43] there are other docs at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#Connecting_to_the_database_replicas_from_your_own_computer [00:19:47] interesting, that's new for me and i am happy to find it out. i wanted remote db access for a long time and for the same long time i was told it is not possible [00:21:28] harej: take a look at those wikitech docs and shout for help if you get stuck [00:22:08] ssh tunnels are the trick and you either use some app that knows about doing that or you do tricks with port mapping locally [00:45:33] bd808: so I'm trying to set up an SSH tunne; [00:45:41] ssh -N -R 8888:enwiki.labsdb:3306 harej@tools-login.wmflabs.org [00:45:44] doesn't seem to be working [00:45:56] or it is, and I am using it wrong [00:46:17] Try using -L instead of -R [00:46:29] and then I would connect to it by doing localhost:8888 [00:46:36] correct [00:50:12] Wow, it actually works. SSH tunnels are witchcraft. Thank you! [00:50:59] The difference between -L and -R is that -L listens locally and forwards to the remote network. -R listens on the remote host and forwards locally [00:51:47] also, you know if it's possible to delete old oauth consumers? [00:52:14] not really, but I could disable them for you I suppose [00:52:46] they're not really doing anything, but it's annoying that they're taking up precious menu space [00:52:52] we should really make it possible for an owner to disable them themsleves [00:52:57] heh [00:56:58] is there a parameter to set up the tunnel and then dump you back to the prompt? [00:59:53] something like: ssh -f -N -L 8888:enwiki.labsdb:3306 harej@tools-login.wmflabs.org [01:00:02] thanks! [01:19:41] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:02:58] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: newer npm for nodejs Kubernetes instances - https://phabricator.wikimedia.org/T169451#3398652 (10Paladox) See https://gerrit.wikimedia.org/r/#/c/303370/ [02:29:40] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [02:45:40] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:20:40] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [03:34:21] bd808: you beat me by 3 mins :P [03:34:55] the curl|sudo or something else? [03:35:24] Patch Set 8: (2 comments) [03:35:34] yeah [03:47:32] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:08:52] 10Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#3398758 (10Beetstra) @valhallasw Do you mind clearing the instance that linkwatcher is on .. it does not have enough resources and starts to build up a backlog. Thanks! [04:17:29] RECOVERY - Puppet errors on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [04:26:49] 10Labs, 10Tool-Labs: Linkwatcher spawns many processes without parent - https://phabricator.wikimedia.org/T123121#1921762 (10bd808) >>! In T123121#1925019, @valhallasw wrote: > Another option, as @Legoktm suggested, is to move LiWa to it's own host. This might be the best option if rewriting the bot to make ef... [04:31:47] 10Labs, 10Tool-Labs, 10Patch-For-Review: Install MP3 encoders and decoders on Tool Labs - https://phabricator.wikimedia.org/T168128#3398768 (10bd808) 05Open>03Resolved a:03valhallasw ``` $ which lame /usr/bin/lame ``` [04:35:57] 10Labs: Blacklist apache from unattended-upgrades on tools puppetmaster - https://phabricator.wikimedia.org/T159254#3061896 (10bd808) Would pinning apache to our repo for jessie in the Puppet class handle this? [05:04:55] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:39:54] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [05:41:40] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:21:43] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [06:28:22] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:45:25] PROBLEM - Puppet errors on tools-exec-1421 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:03:22] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:23] RECOVERY - Puppet errors on tools-exec-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:35] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:47:42] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [08:24:36] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [08:52:41] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [08:58:33] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Move kubernetes authentication to using X.509 client certs - https://phabricator.wikimedia.org/T144153#2590052 (10valhallasw) Are there any turnkey solutions for certificate management? Because generating the keypair is the easy part -- signing (and returning the sign... [10:43:39] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:23:39] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:47] Need help with a node JS application on Tool Labs. [11:39:26] The application is cloned, and the binary is working fine, when I run through terminal. [11:39:33] However I need to automate it. [11:45:48] So, I was saying, the exit status in the error log was 134, "FATAL ERROR: v8::Context::New() V8 is no longer usable" [11:46:47] Now, I did not follow any of these steps (I was told yesterday, that I don't need to do this, but I doubt it) https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#node.js_web_services [11:47:47] Anyone's NodeJS application running on Tool Labs? [11:49:49] me [11:50:26] are you trying to run a webservice? [11:52:09] acagastya: ^ [11:52:43] zhuyifei1999_: It is an application to bridge Telegram and IRC. [11:55:17] * zhuyifei1999_ only run node webservice, not bot, so not too experienced in this [11:55:40] are you trying to run it in grid or k8s? [11:55:55] zhuyifei1999_: what does your application do? [11:57:47] my node webservices? one is an log.io instance that simply broadcasts the logs to brwosers so I don't have to ssh into tool labs to view the logs. another is a socket.io endpoint of another webservice [11:57:59] * another python webservice [11:58:53] (to avoid confusion, my log.io I mean logio.org) [12:00:21] zhuyifei1999_: My application does something similar, it relays messages from IRC to Telegram and vice versa. [12:00:34] How did you do it. [12:00:35] are you trying to run it in grid or k8s? [12:00:41] I mean log io [12:01:50] I did log.io in k8s, but I never ran a bot on k8s. bd808 gave you some instructions at https://phabricator.wikimedia.org/T164910#3271720 if you want to run a node bot on k8s [12:02:27] ks8. [12:02:42] But, yesterday, a user suggested me jstart. [12:02:43] (I basically considers those applications that does not have a web endpoint a bot) [12:02:49] jstart is grid [12:03:14] I was pretty confused, and things did not work. [12:05:15] things to consider on grid: old node version by default (you might want nvm or something similar to install a newer version of node), and low memory allocation by default (jstart -mem 1G will increase the allocation to 1G) [12:06:11] Do I have to use grid? [12:06:29] My application should be always online. That is the only condition. [12:07:21] k8s is next-gen :) [12:08:04] grid is ((very) slowly) passing into history of tool labs [12:09:18] both k8s and grid can be configured to try their best to make sure the application is always online [12:09:39] (in the case of grid, it's jstart or jsub -continuous) [12:10:40] You have done it in k8s, so I think I should go with that. [12:10:51] zhuyifei1999_: So, what should I do? [12:11:10] https://phabricator.wikimedia.org/T164910#3271720 [12:13:43] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Kubernetes#Kubernetes_continuous_jobs [12:14:25] "continuous" means ensure always online [12:14:43] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:14:45] or restart after exit (eg. error) [12:19:26] I will have a look, and get back to you, zhuyifei1999_. [12:19:44] k [12:32:40] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:07:38] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [13:54:40] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [15:31:26] zhuyifei1999_: thanks for pointing acagastya towards https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Kubernetes#Kubernetes_continuous_jobs [15:32:57] I feel like I should write a couple of "my first X" pages about nodejs webservices and bots. [15:33:35] they seem pretty popular but the docs are kind of scattered [15:40:10] yeah, node is really growing [15:43:00] actually, thinking about it, it might be even possible to write a template for my first {python,node,ruby,php,blah} {continuous job,webservice,cronjob} on {kubernetes,grid} guide [15:44:13] yeah, I think they should have a similar structure. The common parts should really be shared too [15:44:24] actually, why not dynamically create the guide based on user input? something like https://tools.wmflabs.org/relgen/ [15:48:14] that could be nice for some of the custom parts yeah. like to make the code samples easier to copy and paste [15:48:56] moving the docs out of the wiki makes keeping them collaboratively updated harder [15:49:13] hmm yeah [15:49:15] maybe there could be something that worked with the wiki though [15:50:13] like a gadget that make the interactive stuff happen with javascript on the client [15:52:59] a tool could also pull the text from wikitech [15:53:21] the gadget would have to interact with the api anyways [15:53:22] *nod* [16:23:44] 10Labs, 10Tool-Labs, 10Documentation: Unite tool labs "my first" tutorials - https://phabricator.wikimedia.org/T169462#3399041 (10zhuyifei1999) [17:15:41] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:50:40] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [19:46:40] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:19:58] 10cloud-services-team, 10Operations, 10Upstream: New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS - https://phabricator.wikimedia.org/T169290#3399312 (10Nemo_bis) [20:21:39] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [20:23:18] 10cloud-services-team, 10Operations, 10Upstream: New anti-stackclash (4.9.25-1~bpo8+3 ) kernel super bad for NFS - https://phabricator.wikimedia.org/T169290#3393693 (10Nemo_bis) "A total of 16,214 non-merge changesets were pulled into the mainline repository for the 4.9 development cycle, making this cycle t... [20:47:40] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:25:36] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:55:37] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0]