[00:18:02] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [01:42:13] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:22:15] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [03:29:05] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:52:55] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:04:04] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [04:12:55] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [05:03:50] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:38:10] 10Tool-Pageviews, 10Hindi-Sites: hi.wikiversity.org is not a valid project error on Pageviews Tool - https://phabricator.wikimedia.org/T176046#3612659 (10Jayprakash12345) > Here's the main page: > http://tools.wmflabs.org/pageviews/?project=hi.wikiversity.org&pages=विकिविद्यालय:मुखपृष्ठ > It seems that Firs... [05:38:50] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:54] 10Tool-Pageviews: Version asset files - https://phabricator.wikimedia.org/T176071#3612673 (10MusikAnimal) [05:46:57] 10Tool-Pageviews, 10Hindi-Sites: hi.wikiversity.org is not a valid project error on Pageviews Tool - https://phabricator.wikimedia.org/T176046#3612687 (10MusikAnimal) >>! In T176046#3612659, @Jayprakash12345 wrote: >> Here's the main page: >> http://tools.wmflabs.org/pageviews/?project=hi.wikiversity.org&page... [05:50:00] 10Tool-Pageviews, 10Hindi-Sites: hi.wikiversity.org is not a valid project error on Pageviews Tool - https://phabricator.wikimedia.org/T176046#3612694 (10Jayprakash12345) >>! In T176046#3612687, @MusikAnimal wrote: >>>! In T176046#3612659, @Jayprakash12345 wrote: >>> Here's the main page: >>> http://tools.wmf... [06:39:31] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:19:33] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:04] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:48:56] 10VPS-project-Wikistats: kbp.wp and din.wp not found in xml file - https://phabricator.wikimedia.org/T176076#3612822 (10Xqt) [10:52:41] 10VPS-project-Wikistats: Add fi.wikivoyage to wikistats - https://phabricator.wikimedia.org/T176077#3612839 (10Xqt) [10:53:09] 10VPS-project-Wikistats: Add fi.wikivoyage to wikistats - https://phabricator.wikimedia.org/T176077#3612854 (10Xqt) [10:55:20] 10VPS-project-Wikistats: kbp.wp and din.wp not found in xml file - https://phabricator.wikimedia.org/T176076#3612857 (10Xqt) Btw: seems they are listed in the csv file. [11:05:03] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:21] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3613078 (10Liuxinyu970226) >>! In T172065#3593626, @Liuxinyu970226 wrote: > @zhuyifei1999 >> https://tools.wmflabs.org/intuition/: https://tran... [15:28:01] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3613112 (10zhuyifei1999) >>! In T172065#3613078, @Liuxinyu970226 wrote: > or I can feel free to create a subtask for this? go ahead. [15:49:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:59:22] PROBLEM - Puppet errors on tools-paws-worker-1016 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:24:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:24] RECOVERY - Puppet errors on tools-paws-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:22] !help. As a part of outreachy microtask for the task Improvements for the Toolforge 'webservice' command.Read over the source code and try to understand what scripts/webservice does when a user runs it as: webservice --backend kubernetes start. I tried running this command but got error as webservice command not found. [17:19:43] Can someone please guide me how to run the script. [17:20:25] Mridu: where are you running the webservice command [17:21:27] I went into tools-webservice then scripts folder. And then ran the command. Using cmd. [17:24:54] Mridu: on your laptop? [17:26:35] Yes [17:36:21] Mridu: so for the command not found thing, you'd need to run something like ./webservice so your terminal can find it in the current directory the first place [17:36:32] in the first place* [17:36:56] which probably still won't work, you might need to get some local development environment set up [17:37:06] I guess there's no instructions on how though [17:37:33] local development enviornment as in? [17:42:06] Mridu: do you have access to Toolforge yet? [17:42:18] I went through https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web. python(python3+kubernetes). Wasn't able to comprehend much. [17:43:09] Don't know. I have created account on wikitech, mediawiki only. Have done nothing with toolforge. [17:43:50] Mridu: right, okay - lets start there - https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Toolforge_users [17:43:55] you can skip step 1 [17:44:11] since your wikitech account is your LDAP account [17:44:18] and you've already created that [17:44:57] yes. [17:46:18] which means ssh -i ~/.ssh/id_rsa username@login.tools.wmflabs.org is the next step. I guess. [17:46:36] uh have you done step 2, filing an access request? [17:47:10] No [17:47:28] Mridu: please do [17:48:24] In reason what I have to mention? [17:49:35] Would mentioning project title be fine [17:50:19] Mridu: something along the lines of access for working on your microtask for outreachy would do [17:52:57] As a part of outreachy project Improvements for the toolforge 'webservice' command. Access for working on the microtask that is to read and understand scripts/webservice. [17:53:30] sure [17:54:44] membership request submitted it shows. [17:55:43] Am I supposed to click on Join Now button. [17:57:01] I approved it, you should have gotten an email or some notificaiton [17:59:10] yes, recieved just now. [18:01:04] cool, now you can check if you can login as described at https://wikitech.wikimedia.org/wiki/Help:Getting_Started#After_account_creation [18:02:20] Generating the ssh key is the 3rd point mentioned. [18:02:57] On clicking the link I get re-directed to the page which I used for setting gerrit. [18:03:10] Mridu: Yeah I assumed you'd done that because you set it up for gerrit [18:03:25] yes [18:03:41] Mridu: is your key already listed here? https://toolsadmin.wikimedia.org/profile/settings/ssh-keys [18:04:40] Do I need to add the same one that I had added in gerrit? [18:05:08] No, it's showing add ssh-key [18:06:21] Mridu: do you have one listed here? https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack [18:08:50] Where are these listed? I am not able to see any ssh-key on this one though. [18:09:23] are you logged in? [18:09:54] if so, go ahead and add your public key in either of these places [18:10:02] (if it's not already there) [18:10:38] Yes, On going into openstack tab. There is a option for ssh-key. It's not listed. Will add. [18:10:42] okay [18:16:26] added it on toolforge. [18:17:03] cool, are you able to login? [18:17:16] On adding same to wikitech it shows failed to import key pair. [18:17:31] either of these places should be fine [18:18:06] I am already logged in to toolforge [18:18:18] cool [18:18:22] should I logout and login again to check? [18:19:39] upto you [18:19:47] you can now try to make a tool account [18:19:52] more details here - https://wikitech.wikimedia.org/wiki/Help:Toolforge#Tool_Accounts [18:21:16] if that all makes sense, you can try following this tutorial to make a webservice and run it using the kubernetes backend https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool [18:22:11] By the end of step 2 you should have a running webservice, and some better understanding for what it does [18:23:15] unable to see "create new tool link" [18:23:41] https://toolsadmin.wikimedia.org/tools/ [18:23:47] on the left side bottom [18:25:19] on left I can only see join now and membership requests. [18:25:35] relogin? [18:26:15] Have not relogined. Will do. [18:27:05] yes. It's there. [18:28:27] !log tools.heritage Restarted the webservice − Yarl notified us around 10 hours ago that API was inaccessible. `webservice restart` fixed the issue. [18:28:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [18:28:43] !log tools.heritage Deploy latest from Git master: b3c3ab0 (T174871) [18:28:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [18:28:48] T174871: Solve ErfgoedBot categorisation problem - https://phabricator.wikimedia.org/T174871 [18:32:05] madhuvishy: So, once I make this My_first_Flask_OAuth_tool. I would be able to run the webservice command? [18:33:19] And kubernetes would be running in backend. [18:33:45] uh, don't make it literally 'My_first_Flask_OAuth_tool' [18:33:54] but yes [18:34:06] after you `become ` [18:35:58] * zhuyifei1999_ imagines a few years later we lave a ton of duplicate my_first_blah_tool test tools created by those following the tutorials... [18:37:34] haha. yes! I mentioned it as a example. :) [18:37:42] Mridu: so Toolforge is an environment in which we provide abstractions for all kinds of applications. We have one off jobs, continuous jobs, webservices, bots etc that run there. the `webservice` command is a script that lets folks make a web app, be it a simple static webpage or a flask app or a node app, and run it in our environment. We have a bunch of servers put together that provide resources for your webservice to [18:37:43] run, so we need some way to take your webservice and run it somewhere where we can fit it (cpu/memory/etc). Kubernetes allows us to do just that. So we have a kubernetes cluster running in toolforge, and when you run `webservice --backend=kubernetes python start` the Webservice command wraps up your app in a little container thing and ships it off to be run on k8s. [18:37:48] k8s = kubernetes [18:38:25] We have other little helpers to run one off and continuous jobs (jsub is one of them) [18:41:53] It's hard to replicate what webservice does on your laptop since you don't have a k8s cluster running where you can schedule things, you would have to set up a bunch of things to get it running - that would be your development environment. But I'd set that aside for now and just use a simple example tool and the toolforge environment to try and understand how webservice works for your microtask [18:43:08] yes. I am trying to grasp this. Sorry for a trivial question. But, can you please give an example of "job"? [18:43:54] Mridu: sure, a job can be any command that you can run on unix, from `ls` to `python mybot.py` [18:47:14] ok. "so Toolforge is an environment in which we provide abstractions for all kinds of applications". By abstractions do you mean providing the required enviornment for the app to run? [18:49:53] yes exactly. if you had to deploy a flask webservice in a server somewhere, you'd need to set up a webserver like uwsgi and make sure it keeps running, restart it if it dies, etc. Toolforge manages all that for you, it lets you just write your code, run something like `webservice start` and then forget about it [18:49:56] for the most part [18:52:37] "deploy a flask webservice in a server somewhere" - does that mean hosting the flask web app? [18:55:10] Yup [18:56:17] Different applications would need different environment. How do we manage that? [19:01:59] you specify it [19:02:20] eg. `webservice --backend=kubernetes python start` python specify a python environment [19:02:36] and uses your virtualenv for python packages [19:07:49] okay.. able to get something. And role of kubernetes? [19:11:49] k8s runs the service [19:12:26] it starts and manages 'containers' that runs the webservice process [19:12:52] webservice process may be a uwsgi process, or a nodejs process, for example [19:13:28] 10Toolforge: Install nmap on Toolforge - https://phabricator.wikimedia.org/T172688#3613326 (10Huji) 05Open>03declined For reasons mentioned by bd808 [19:13:33] the container and process is specified via 'python' [19:14:00] python => uses a container with python & uwsgi installed, and starts the uwsgi process [19:15:48] webservice? [19:16:06] the backend could also be 'gridengine' but it's semi-obsolete and I think you can ignore that [19:16:08] ? [19:16:23] webservice s the command that controls webservices... [19:16:30] *is [19:16:58] Sorry for asking it the wrong way. I meant meaning or example of a webservice. [19:17:46] a webservice is a service that listens on a port and responds with HTTP protocol [19:18:09] the port may be arbitrary though, so don't hardcode port 80 [19:18:41] in toolforge a webservice is limited to a path, not the full domain [19:19:14] i.e. a webservice can only respond to those under tools.wmflabs.org/ [19:19:36] oh.. okay. [19:20:11] there's nginx router (if I remember correctly) that routes different paths to different webservices [19:20:37] btw, if I say 'IIRC' or 'iirc' I mean 'if I remember correctly' [19:21:59] so toolforge basically has two backends. That is grid engine and kubernetes. And kubernetes helps toolforge provide the environment required by various web apps(tools present on toolforge). [19:22:28] kind of, but grid does that as well, but you can ignore that [19:23:53] multiple apps would need different environment. So, with the help of kubernetes and grid engine we provide that. [19:24:09] kind of [19:24:35] in the kase of k8s environment is in containers [19:25:32] containers are created from 'images', which are created from dockerfiles in https://github.com/wikimedia/operations-docker-images-toollabs-images/ [19:25:50] apps having different env would have diff containers. [19:26:24] according to the requirements that are needed for deploying the app. [19:26:46] yes, a node webservice, a python webservice, and a php webservice all use differerent containers [19:27:19] I don't know the exact definition of a 'container' though [19:27:27] wow, all this sounds interesting. Thanks. [19:27:59] btw, if you see 'toollabs' or 'tool labs' we mean toolforge [19:28:09] it's the old name [19:30:04] yes. I read somewhere. [19:38:56] Thanks for clarifying things. :) [19:41:52] np [19:54:45] Hii ! I created an account on Toolforge and initiated a request for being a member of toolsforge so that I can create a tool for user contribution summary . But , when I went on create a tool , it shows me 403 Forbidden status Code , where as my request has been approved , Request ID a294d3bb42984173927349b2877c4f83 [19:55:20] 10Cloud-Services, 10Gerrit, 10Repository-Admins: Deactivate repository labs/invisible-unicorn - https://phabricator.wikimedia.org/T154099#3613356 (10MarcoAurelio) [19:58:41] bd808: ^ [20:22:04] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:07:04] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:54] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3613434 (10Krenair) >>! In T174860#3596419, @Andrew wrote: > Managing this domain will be a bit of a pain, since it's in noauth-project. [... [21:55:23] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3613435 (10Krenair) I am curious as to what your designateclient hack is though [22:36:32] !log tools.heritage Service down again ("backend is overloaded" in error.log). Tried another webservice restart [22:36:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [23:19:12] (03PS1) 10Lokal Profil: [WIP] Ensure skipped image categorizations are mentioned in stats [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378641 (https://phabricator.wikimedia.org/T174871) [23:20:42] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Ensure skipped image categorizations are mentioned in stats [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378641 (https://phabricator.wikimedia.org/T174871) (owner: 10Lokal Profil)