[11:16:48] <jynus>	 fwd: <jynus> !log stopping db1064 for cloning to db1121 (will create temporary lag on commons wikireplicas)
[11:17:11] <jynus>	 (it should only take around 1h30 or so to be back in a good state)
[11:51:56] <annika>	 why is there no linking to ssh host fingerprints from help:access and why are the tools fingerprints obsolete??
[12:02:34] <arturo>	 annika: this page? https://wikitech.wikimedia.org/wiki/Help:Access
[12:03:34] <annika>	 yes
[12:04:27] <arturo>	 let's fix that! :-)
[12:05:34] <arturo>	 annika: so you are referring to 2 different things. First, obsolete fingerprints? where? please share link
[12:13:31] <annika>	 https://wikitech.wikimedia.org/wiki/SSH_Fingerprints or similar is not linked and https://wikitech.wikimedia.org/wiki/Obsolete:SSH_Fingerprints/tools-login.wmflabs.org is obsolete, but i see that it is the old domain name in the title
[12:29:53] <arturo>	 annika: `tools-login.wmflabs.org` still works as login FQDN
[12:47:35] <maxlath[m]>	 hi, I'm trying to create a new tool, but any name starting by `wikidata` (or actually any wikimedia project) is rejected, while I see existing tools with those names: is there an authorization level required to create a tool with this kind of name?
[12:48:16] <chicocvenancio>	 maxlath[m]: no, the name is checked from the global name filter
[12:48:58] <chicocvenancio>	 and all projects were readded there some time after those `wikidata*` tools were created
[12:52:49] <maxlath[m]>	 so how does that work? no project name can contain another project name?
[12:54:18] <chicocvenancio>	 maxlath[m]: https://phabricator.wikimedia.org/T190707
[12:54:40] <chicocvenancio>	 no username can contain a project name
[12:58:20] <maxlath[m]>	 thanks
[13:33:29] <jynus>	 commons should be almost back to normal: https://tools.wmflabs.org/replag/
[13:34:34] <Hauskatze>	 what's up with s4?
[13:35:35] <jynus>	 Hauskatze: regular maintenance, we are upgrading production servers
[13:35:53] <jynus>	 that creates temporary disruption on the replication
[13:48:55] <maxlath[m]>	 hi again, is there a way to access dumps on /mnt when logged in a nodejs shell (`webservice --backend=kubernetes nodejs shell`)? I would like to access /public/dumps/public/wikidatawiki/entities/latest-all.json.gz but in this mode I get:
[13:48:56] <maxlath[m]>	 `/public/dumps/public: broken symbolic link to /mnt/nfs/dumps-labstore1006.wikimedia.org/xmldatadumps/public`
[14:14:58] <chicocvenancio>	 maxlath[m]: That seems like a bug. Can you create a phabricator task?
[14:44:44] <Nudin_WMDE>	 Technical Advice IRC meeting starting in 15 minutes in channel #wikimedia-tech, hosts: @Tonina_WMDE & @Lucas_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[14:44:44] <maxlath[m]>	 chicocvenancio: created here https://phabricator.wikimedia.org/T193646 let me know if it needs any other details
[17:35:15] <greg-g>	 is there an automagically created set of monitoring graphs of eg disk/cpu for all hosts in cloud projects? My issue is I want to see which jenkins permanent hosts are almost full disk
[17:37:11] <chasemp>	 greg-g: everything should publish to graphite
[17:37:33] <greg-g>	 url?
[17:37:40] <greg-g>	 graphite.wmflabs?
[17:37:45] <framawiki>	 andrewbogott: projects for https://phabricator.wikimedia.org/T193653 ? :)
[17:38:07] <chasemp>	 greg-g: https://graphite-labs.wikimedia.org/
[17:39:05] <chasemp>	 greg-g: example url https://graphite-labs.wikimedia.org/render/?width=586&height=308&_salt=1525282727.824&target=deployment-prep.deployment-apertium02.diskspace.root.byte_avail&target=deployment-prep.deployment-apertium02.diskspace.root.byte_used&areaMode=stacked
[17:40:37] <andrewbogott>	 framawiki: I don't understand your question, sorry
[17:42:08] <chasemp>	 greg-g: need help?
[17:43:34] <greg-g>	 we don't have a fancy frontend that automatically makes nice dashboards, right?
[17:46:02] <chasemp>	 greg what's the project?
[17:46:21] <chasemp>	 I think we might have something more curated but I'm unsure if it's default
[17:47:13] <chasemp>	 greg-g: anyway here's an examle https://tools.wmflabs.org/nagf/?project=xtools
[17:47:32] <greg-g>	 chasemp: integration
[17:48:29] <greg-g>	 I think that's what I was looking for :)
[17:48:46] <greg-g>	 except the diskspace free graph for integration is corrupted :/
[17:58:35] <paladox>	 greg-g there's https://grafana-labs.wikimedia.org/?orgId=1
[17:58:57] <paladox>	 https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?orgId=1
[18:04:52] <zhuyifei1999_>	 !log quarry Deployed f4e86f1 and restarted everything
[18:04:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[18:06:27] <zhuyifei1999_>	 Steinsplitter: ^
[18:06:46] <Steinsplitter>	 thanks 
[18:23:32] <framawiki>	 !log quarry deployed af2f7e6 to quarry-main-01
[18:23:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[18:38:08] <zhuyifei1999_>	 framawiki: nice
[18:39:22] <zhuyifei1999_>	 framawiki: https://quarry.wmflabs.org/run/257422/output/1/html?download=true <= 502
[19:48:06] <srish_aka_tux__>	 Hello! I have a tool web page related question. For my tool (/gerrit-newcomer-bot/), while the web service is running and the tool is doing its thing, I am not able to show a webpage for it. As per the instructions in  https://wikitech.wikimedia.org/wiki/Help:Toolforge#Creating_a_Tool_web_page, I dropped an index.html file in the public_html directory. Though, .description file is working fine. Is there
[19:48:06] <srish_aka_tux__>	 anything else that I am missing? 
[19:49:43] <chicocvenancio>	 hey srish_aka_tux__ , how does it do its thing?
[19:52:31] <srish_aka_tux__>	 @chicocvenancio Hey! so the code for the tool (https://github.com/srish/gerrit-newcomer-bot) is in /www/python/src, and it is running the script fine. But, I want to show some description of the tool on its webpage. 
[19:53:48] <chicocvenancio>	 are you running it with `webservice`?
[19:54:11] <srish_aka_tux__>	 Yes! I'm. Following the steps in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python_(Python3_+_Kubernetes). 
[19:54:34] <chicocvenancio>	 it seems to me it is not a web tool, I'm reading that wrong?
[19:55:11] <srish_aka_tux__>	 That's right, it is not a web tool. 
[19:56:10] <srish_aka_tux__>	 Though, I want to show the details of the tool here: https://tools.wmflabs.org/gerrit-newcomer-bot
[19:57:11] <chicocvenancio>	 then it shouldn't be started with `webservice`, that command tells the proxy to route the tool address to the kubernetes pod. 
[20:16:12] <zhuyifei1999_>	 srish_aka_tux__: for your bot, you are making a 'script' that basically checks for any new people joining gerrit then sends a message to them right?
[20:16:45] <zhuyifei1999_>	 and there will not be anyone who proactively visit tools.wmflabs.org/gerrit-newcomer-bot ?
[20:17:07] <zhuyifei1999_>	 oh you want to show the details in a web page
[20:17:19] <chicocvenancio>	 we discussed leaving it like this for now
[20:18:07] <chicocvenancio>	 (I do think we should have a command for this use case, as simple as webservice)
[20:18:27] <srish_aka_tux__>	 @zhuyifei1999_ yes, that's right! 
[20:18:44] <zhuyifei1999_>	 chicocvenancio: I think the lighttpd webservice does that
[20:18:50] <zhuyifei1999_>	 lighttpd on grid
[20:19:22] <zhuyifei1999_>	 srish_aka_tux__: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Default_web_server_(lighttpd_+_PHP)
[20:19:29] <chicocvenancio>	 grid is a simpler than k8s right now, I agree
[20:19:49] <chicocvenancio>	 but the hard part is moving from webservice to manual k8s deployment for the bot part
[20:20:41] <bd808>	 the only real docs on that today are at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_continuous_jobs
[20:20:42] <srish_aka_tux__>	 @zhuyifei1999_ the bot is written in Python, so I am guessing that link won't work? 
[20:21:14] <zhuyifei1999_>	 srish_aka_tux__: you are showing only an index.html right?
[20:21:47] <srish_aka_tux__>	 Yes, that's correct! 
[20:22:00] <chicocvenancio>	 it will work for the webpage
[20:22:03] <zhuyifei1999_>	 (we should really fix make deploying bots on k8s easier)
[20:22:15] <chicocvenancio>	 but python is now running on the k8s "web" pod
[20:22:17] <bd808>	 the lighttpd config that the python webservice backend sets up only routes requests to a wsgi app.  I'm not sure if it would be easy to add a ~/.lighttpd.conf that would server static pages or not
[20:22:19] <zhuyifei1999_>	 srish_aka_tux__: so lighttpd will serve that single webpage
[20:23:20] <bd808>	 stashbot handles this by using a manual pod for the bot and then webservice for the http page (which is really just a static stub)
[20:23:21] <stashbot>	 See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help.
[20:23:43] <chicocvenancio>	 zhuyifei1999_: Can't we leave the pod working as is for now (without a webpage) and migrating this latter?
[20:24:45] <zhuyifei1999_>	 ummm... I don't think a bot in a 'web' pod makes sense... tools.wmflabs.org/gerrit-newcomer-bot times out
[20:25:08] <zhuyifei1999_>	 but I think it's srish_aka_tux__'s option
[20:25:20] <chicocvenancio>	 indeed, but it does work...except for the timeout
[20:25:45] * chicocvenancio is very lazy today
[20:26:16] * zhuyifei1999_ should work on the toolsbeta stuffs again, after losing two rounds of chess :(
[20:26:50] * srish_aka_tux__ should dive into this stuff for better understanding :) 
[20:31:01] <zhuyifei1999_>	 !log toolsbeta Added k8s_infrastructure_users to project hiera on horizon T192618
[20:31:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[20:31:03] <stashbot>	 T192618: Can't ssh into new instance created at toolsbeta - https://phabricator.wikimedia.org/T192618
[20:31:32] <zhuyifei1999_>	 !log toolsbeta nuke webservice instances and rebuild
[20:31:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[20:37:52] <bd808>	 !log toolsbeta Added Neha16 as a project admin for work on T175768
[20:37:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[20:37:54] <stashbot>	 T175768: Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T175768
[20:50:58] <zhuyifei1999_>	 bd808: how long does it take for a change in project hiera to be reflected when rebuilding the instance?
[20:52:31] <zhuyifei1999_>	 hiera('k8s_infrastructure_users') still seems to be null while building toolsbeta-proxy-01
[20:53:02] <bd808>	 zhuyifei1999_: it should be functionally instant I think. I don't remember a cache layer that would need to expire
[20:54:36] <chicocvenancio>	 zhuyifei1999_: did you check those roles?
[20:54:40] * zhuyifei1999_ experiments with stuffs
[20:55:11] <chicocvenancio>	 I vaguely remember that few did checks on which hosts they were installed before applying anything
[20:55:27] <chicocvenancio>	 s/few/a few/
[20:56:17] <chicocvenancio>	 as in, they wouldn't "just work"™ in toolsbeta
[20:56:35] * chicocvenancio is sorry I can't help more right now
[20:58:58] <zhuyifei1999_>	 I think I see something weird here https://github.com/wikimedia/labs-private/blob/ef99748e1c4674007bb5d78e363da9c5398ff81f/hieradata/common.yaml#L50 https://github.com/wikimedia/labs-private/blob/35ae7b8dad4e832d197f618b4beaa9fb1f34fd94/hieradata/labs/tools/common.yaml#L4 are both set
[20:59:21] <zhuyifei1999_>	 do they merge aotomatically?
[21:00:17] * chicocvenancio is busy learning ansible, looking at puppet doesn't make sense atm
[21:00:35] <bd808>	 zhuyifei1999_: the second one would apply in the tools project. The first would apply in other projects in Cloud VPS
[21:00:47] <zhuyifei1999_>	 I think the latter might have override the former and it is the ['token'] part $users['proxy-infrastructure']['token'] failed
[21:00:49] <zhuyifei1999_>	 ok
[21:01:36] <zhuyifei1999_>	 I guess I'll just copy that over to toolsbeta
[21:02:11] <chicocvenancio>	 I think it makes sense to create new roles for toolsbeta
[21:02:43] <chicocvenancio>	 hopefully inheriting from the toolforge stuff, if it is possible
[21:02:51] <zhuyifei1999_>	 !log toolsbeta copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
[21:02:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[21:18:31] <Hauskatze>	 what's the 'pif_edits' table for?
[21:19:28] <legoktm>	 https://phabricator.wikimedia.org/T54921 doesn't say
[21:19:33] <legoktm>	 sounds like personal image filter tbh
[21:20:56] <Hauskatze>	 well querying the table doesn't give much info either
[21:21:11] <Hauskatze>	 pif_user and pif_edits are the columns
[21:22:43] <legoktm>	 10:52 Andrew: Running populatePifEditCount.php on all wikis
[21:22:43] <legoktm>	 10:45 Andrew: Adding pif_edits table to all wikis for personal image filter voter list
[21:22:46] <legoktm>	 via https://wikitech.wikimedia.org/wiki/Server_admin_log/Archive_19
[21:23:00] <Hauskatze>	 pff, that's *old*
[21:23:08] <legoktm>	 so from the referendum
[21:23:12] <Hauskatze>	 yeah
[21:23:35] <Hauskatze>	 the result was somewhat the community wanted it but it was never implemented, but my memory can fail
[23:18:18] <zhuyifei1999_>	 chicocvenancio: a lot of the services refuse to start. nginx, etcd, kube-apiserver, blah blah blah, could you look into it when you get time?
[23:18:56] <chicocvenancio>	 sure, but it'll probably be saturday
[23:19:48] <zhuyifei1999_>	 ok
[23:26:52] <zhuyifei1999_>	 etcd is failing because it tries to connect to peers
[23:27:08] <zhuyifei1999_>	 (which consists of only itself)
[23:29:01] <zhuyifei1999_>	 let's see if I can get grid to work