[00:18:29] <andrewbogott>	 ebernhardson: that got us to the next bug :) 
[00:18:30] <andrewbogott>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, $cluster_name must not be set to "elasticsearch" at /etc/puppet/modules/elasticsearch/manifests/instance.pp:147:9 at /etc/puppet/modules/elasticsearch/manifests/init.pp:113 on node
[00:18:30] <andrewbogott>	 deployment-elastic07.deployment-prep.eqiad.wmflabs
[00:19:47] <ebernhardson>	 umm, hmm i didn't realize we did that. The upstream default is elasticsearch
[00:21:21] <ebernhardson>	 andrewbogott: i suppose i should have been asking, why doesn't this cluster have a name? Other cloud instances probably don't have names, but deployment-prep reports as being a member of beta-search.  So it means the default cluster name needs a decision, and i have to figure out why the beta-search name didn't make it through
[00:30:43] <ebernhardson>	 I might have to leave this for gehel :S Looking at deployment-prep's hieradata and the elasticsearch profile and module ... at least running the code in my head the name ends up in the right place :S
[00:31:01] <abartov>	 zhuyifei1999_: thanks!  That seems like a relevant issue.  Apparently abandoned since 2016?  This seems like a barrier to core content, and therefore should be high priority, no?
[00:32:08] <zhuyifei1999_>	 you mean the 503 error?
[00:32:34] <zhuyifei1999_>	 you just need async chunked uploading. videoconvert doesn't support that
[00:47:29] <harej>	 abartov: does https://tools.wmflabs.org/video2commons/ do the thing you need to do, or does videoconvert do something else?
[00:48:45] <abartov>	 No, it does not, unless I upload my cut MP4 files to YouTube first.
[00:49:18] <abartov>	 I suppose I'll have to convert to webm locally with ffmpeg and upload durectly
[00:49:28] <abartov>	 Directly, even.
[00:50:16] <zhuyifei1999_>	 v2c allows you to upload files to the tool
[03:42:32] <abartov>	 zhuyifei1999_: oh! I didn't realize that! Thanks!
[08:36:55] <joal>	 Hi Cloud chan
[08:37:17] <joal>	 Yesterday I managed to get some queries run against quarry DB, but got suprisingly unexpected results
[08:37:45] <joal>	 Shall I wait for zhuyifei1999_ to get some answer or is there somebody else having some knowledge of the thing?
[08:37:52] <joal>	 Many thanks in advance :)
[08:38:31] <zhuyifei1999_>	 joal: I’m supposed to be sleeping right now, but I guess I can do
[08:38:54] <joal>	 zhuyifei1999_: Please get some sleep if needed, I'm not in a rush preventing people to sleep :)
[08:39:19] <zhuyifei1999_>	 oh I have sleep troubles so it barely matters
[08:39:32] <joal>	 zhuyifei1999_: Arf :( Good luck with that :/
[08:39:55] <joal>	 zhuyifei1999_: My understanding of the run table is that it would hold data for all previous queries
[08:40:03] <zhuyifei1999_>	 yes
[08:40:13] <joal>	 but from the result you sent me, it doesn't seem vaaid
[08:40:16] <joal>	 valid sorry
[08:40:30] <zhuyifei1999_>	 why not?
[08:41:06] <joal>	 For instance: count by status gives us a result only for status 0 - Given we know some queries are killed, well there is a discrepency there
[08:41:32] <joal>	 OH NO ! I think I know why
[08:42:16] <joal>	 My bad, I forgot the group by in the query :S How can I have done that /facepalm
[08:42:54] <zhuyifei1999_>	 lol
[08:42:54] <joal>	 I would have expected the query to fail though - Any way - Do you mind zhuyifei1999_ rerunning this one: SELECT status, COUNT(1) as c FROM query_run group by status;
[08:43:16] <zhuyifei1999_>	 I didn't really read what the queries are actually doing
[08:43:27] <zhuyifei1999_>	 k
[08:43:35] <joal>	 zhuyifei1999_: same exact problem for second query - My super bad :S Not being able to test is not great
[08:43:59] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/wFC8Js1Z/
[08:44:11] <joal>	 YES ! Makes a lot more sense :)
[08:44:28] <joal>	 zhuyifei1999_: I'm editing the queries, will resend the link in a minute
[08:44:31] <zhuyifei1999_>	 and what the second query should be changed into?
[08:44:33] <zhuyifei1999_>	 ok
[08:45:09] <joal>	 zhuyifei1999_: https://gist.github.com/jobar/39e9428a332d9bb2823ff03a6ab213a8
[08:45:23] <joal>	 zhuyifei1999_: With my apologizes for not having realized before :S
[08:45:48] <zhuyifei1999_>	 both second and third query?
[08:46:21] <joal>	 zhuyifei1999_: correct, but I'm also correcting the points you mentionned earlier - Can you reload the page please?
[08:46:37] <zhuyifei1999_>	 k
[08:46:38] <joal>	 Now we should be good (comma removed and space after sum removed as well)
[08:47:14] <zhuyifei1999_>	 you might also want to add a semicolon after the second query
[08:47:24] <joal>	 AHHH - crap - yes
[08:48:02] <zhuyifei1999_>	 the text field is way too large for me to just paste it. can I redact that?
[08:48:33] <joal>	 zhuyifei1999_: My point is in fact to get queries :S Is there an easy way to put that in a file?
[08:48:46] <zhuyifei1999_>	 umm. ok I'll paste
[08:51:38] <zhuyifei1999_>	 I should try to figure out how to save it to, say, json. this is unreadable
[08:52:30] <joal>	 zhuyifei1999_: Mwarf :(
[08:52:42] <zhuyifei1999_>	 nevermind I think I got a tsv by piping the query to its stdin
[08:55:12] <zhuyifei1999_>	 https://usercontent.irccloud-cdn.com/file/lU1ARHLG/for_joal.tsv
[08:55:50] <zhuyifei1999_>	 the third query:
[08:55:56] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/ov4gn684/
[08:56:26] <joal>	 That's awesome zhuyifei1999_ :)
[08:56:37] <zhuyifei1999_>	 np
[08:57:41] <joal>	 zhuyifei1999_: I let you get some sleep now :)
[08:58:18] <zhuyifei1999_>	 I can't believe there isn't one with 10+ runs, so:
[08:58:32] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/MSEVpYjf/
[08:59:08] <zhuyifei1999_>	 I think this query isn't doing what's it intended to
[08:59:24] <zhuyifei1999_>	 oh
[08:59:52] <joal>	 zhuyifei1999_: I think it does, but I had not a good enough understanding of how the thing work when I devised the query
[09:00:07] <zhuyifei1999_>	 ok so a query has multiple revisions, and each revision has a run
[09:00:13] <joal>	 zhuyifei1999_: Indeed !
[09:00:27] <joal>	 zhuyifei1999_: Will try to revamp thequery to get us more insteresting resultas
[09:00:36] <zhuyifei1999_>	 k, shall I wait?
[09:01:13] <joal>	 zhuyifei1999_: Please don't wait, go to bed or do whatever you prefer, I'll paste the query here when ready, and I'll wait for to be available :)
[09:01:20] <zhuyifei1999_>	 k
[09:01:28] <joal>	 Many thanks again - I owe you one :)
[09:01:33] <joal>	 (or more)
[09:14:23] <joal>	 zhuyifei1999_: https://gist.github.com/jobar/ba610e878342d1ccf837e3b54eebd410
[09:14:29] <joal>	 And from that maybe more
[09:43:20] <godog>	 !log tools restart prometheus@tools on prometheus-01
[09:43:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:21:44] <arturo>	 thanks godog 
[10:25:39] <godog>	 np arturo, that was part of me taking a quick look at T209893
[10:25:40] <stashbot>	 T209893: Toolforge tools.tools-k8s-master-01.KubernetesCollector.namespaces.active graphite metric not being updated - https://phabricator.wikimedia.org/T209893
[10:26:32] <godog>	 is tools' puppetmaster going to eventually sync/rebase with puppet.git ?
[10:27:59] <arturo>	 I would say so
[10:28:05] <arturo>	 what is the diff right now?
[10:29:11] <godog>	 essentially without https://gerrit.wikimedia.org/r/c/operations/puppet/+/474866 applied, though not a whole lot of time has passed yet since the merge, I'll try again in 10 min or so
[10:29:48] <arturo>	 let me check
[10:30:47] <arturo>	 godog: rebased, you are all set :-)
[10:31:11] <arturo>	 I'm sure there is a cronjob somewhere, but not sure about the frequency
[10:32:03] <godog>	 sweet, thanks arturo ! yeah that works
[10:32:22] <godog>	 heh it'll be in /var/spool/cron/crontabs I think, if it is managed by puppet
[10:34:18] <arturo>	 :-P
[10:52:23] <arturo>	 !log tools T208579 distributing now misctools and jobutils 1.33 in all aptly repos
[10:52:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:52:27] <stashbot>	 T208579: tools-mail: Migrate to Stretch - https://phabricator.wikimedia.org/T208579
[13:00:41] <refeed>	 Hi, any Quarry maintainer here?
[13:01:41] <Urbanecm>	 refeed, it would be better to ask your question, Quarry maintainer might not be around, but your question might be answerable by more people than just Quarry maintainers. 
[13:01:45] <Urbanecm>	 In another words, don't ask to ask, just ask :)
[13:04:09] <paladox>	 !ask
[13:04:09] <wm-bot>	 Hi, how can we help you? Just ask your question.
[13:05:14] <refeed>	 Urbanecm: Alright, honestly I need help with https://gerrit.wikimedia.org/r/#/c/analytics/quarry/web/+/474532/ , I just wanna ping them if they have any free time to review it :)
[13:09:21] <Urbanecm>	 refeed, I added some reviewers based on https://gerrit.wikimedia.org/r/admin/groups/b992861bb3f5df1d6f3df8b821c0bf919bb713a8,members
[13:09:59] <refeed>	 Urbanecm: Thanks! that helps
[13:10:06] <Urbanecm>	 yw
[13:15:17] <andrewbogott>	 godog: puppet is broken on deployment-redis05 and deployment-redis06 (both of which you created.)  Are they obsolete or should someone try to fix them?
[13:19:13] <godog>	 andrewbogott: no idea tbh, I created them to help with the redis stretch migration IIRC but I'm not involved in redis
[13:19:29] <andrewbogott>	 any idea who might now?
[13:19:33] <andrewbogott>	 *know?
[13:23:26] <godog>	 mhhh probably asking in -operations or _security
[13:24:33] <gtirloni>	 !log tools shutdown tools-clushmaster-01 (use tools-clushmaster-02)
[13:24:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:26:52] <andrewbogott>	 akosiaris and/or Krenair, any chance that deployment-zotero01 can be deleted?  It's Trusty (so needs rebuilding anyway) and also uses a no-longer-present base image so moving it will be a bit dicey.
[13:34:21] <akosiaris>	 andrewbogott: yes, but not right now. Probably in a couple of weeks though
[13:34:37] <akosiaris>	 the replacement is being currently deployed and tested in production
[13:34:54] <andrewbogott>	 ok
[13:36:14] <andrewbogott>	 I'll see if I can salvage the VM then :)
[13:44:59] <andrewbogott>	 mobrovac, can you tell me about deployment-redis VMs in deployment-prep?  They were involved in the jobqueue so I'm guessing they're just leftovers at this point...
[13:45:05] <andrewbogott>	 would like confirmation before I delete them though :)
[13:47:36] <Monkey_>	 Go
[13:48:43] <Monkey_>	 Throw it away
[13:55:45] <andrewbogott>	 !log deployment-prep deleting deployment-redis05 and deployment-redis06 as per Giuseppe, "we're not using the old jobqueue, we should remove those vms"
[13:55:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[14:18:30] <gtirloni>	 !log tools Created Puppet prefixes 'tools-clushmaster' & 'tools-mail'
[14:18:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:48:44] <Adithyak1997>	 !help I have a python file and a html file. I would like to know how I can make the html page to reload every 10 minutes(For eg.https://tools.wmflabs.org/fireflytools/emptypages/enwiki)
[15:48:44] <wm-bot>	 Adithyak1997: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[17:20:26] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/s4Fs6xbX/
[17:20:30] <zhuyifei1999_>	 joal: ^
[17:23:05] <joal>	 thanks zhuyifei1999_ - Obviously, same problem again :( Updated the query: https://gist.github.com/jobar/ba610e878342d1ccf837e3b54eebd410
[17:23:30] <joal>	 zhuyifei1999_: I'll stop bothering you after that, I have interesting data already :)
[17:23:38] <joal>	 Many thanks :)
[17:24:20] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/Wi5sJLgX/
[17:24:25] <zhuyifei1999_>	 it's okay
[17:26:17] <joal>	 Awesome :)
[17:26:49] <zhuyifei1999_>	 joal: just wondering, what project is the data for?
[17:27:58] <joal>	 zhuyifei1999_: The analytics-team (come and talk to us in #wikimedia-analytics :) is willing to add an analytical SQL query engine to labs, with some precomputed data out of mediawiki database
[17:28:33] <zhuyifei1999_>	 k nice
[17:28:37] <joal>	 zhuyifei1999_: the closest of what we'd like to provide is quarry, so I'm trying to get some insights
[19:25:15] <maxlath>	 hi! a process I'm running on Toolforge keeps getting `SIGKILL`ed https://phabricator.wikimedia.org/T209887 , any clue why?
[19:25:55] <zhuyifei1999_>	 looking
[19:27:28] <zhuyifei1999_>	 maxlath: I don't see pid 11634 on tools-bastion-03
[19:30:57] <legoktm>	 maxlath: is your script running via the grid? (jsub)
[20:16:07] <maxlath>	 zhuyifei1999_: it has been killed in the meantime
[20:16:23] <zhuyifei1999_>	 k
[20:16:35] <zhuyifei1999_>	 well, I think it's the CPU time limit
[20:16:46] <zhuyifei1999_>	 so yeah, you better run it on grid
[20:20:08] <maxlath>	 zhuyifei1999_: ok thanks, I will have a look at it
[21:18:51] <zhuyifei1999_>	 matanya: do you have time to rebuild the video encoding* on stretch? /me wants to take advantage of https://phabricator.wikimedia.org/T190333
[21:22:56] <zhuyifei1999_>	 encoding01 & 02 are being depooled (in progress) for a code update. only encoding03 is pooled
[21:23:26] <zhuyifei1999_>	 but with the switch to VP9 encoding takes much longer :(
[21:26:14] <zhuyifei1999_>	 (I could build ffmpeg from source for use in the meantime)
[21:41:20] <zhuyifei1999_>	 !log video building a gentoo prefix on /data/project/gentoo-prefix so v2c can use multi-thread encoding with libvpx-vp9 1.7
[21:41:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[22:02:30] <mutante>	 !log wikistats deleted instance wikistats-cowgirl (jessie), replaced after > 3 years
[22:02:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL
[22:28:07] <mutante>	 i have an instance with a puppet role which tries to deploy with scap.. i want to test changes on cloud
[22:28:19] <mutante>	 for some reason it keeps trying deployment-tin.deployment-prep.eqiad.wmflabs
[22:28:35] <mutante>	 which is outdated. i need to use  deployment-deploy01.deployment-prep.eqiad.wmflabs
[22:29:28] <mutante>	 what i dont get is why.. because deployment-tin: does not appear in entire puppet repo, does not appear in my Hiera config , neither for instance nor project nor prefix..  AND i already set both scap::deployment_server and deployment_server to it in my instance Hiera in Horizon
[22:29:39] <mutante>	 where is deployment-tin still a default?
[22:30:36] <mutante>	 always "failed to resolve deployment-tin.deployment-prep" .. yea..i  know.. but dont use that :)
[22:30:45] * Hauskatze suggests grep for deployment-tin to find where
[22:31:16] <mutante>	 i did. that string is not inside the puppet repo
[22:31:56] <Hauskatze>	 I think Puppet is one of the Holy Misteries
[22:31:59] <mutante>	 looks more in scap repo
[22:32:48] <mutante>	 it's also not in the scap repo ...
[22:33:17] <Krenair>	 mutante, maybe in project puppet config somewhere?
[22:33:27] <Krenair>	 or prefix puppet?
[22:33:37] <Krenair>	 or wikitech?
[22:33:47] <Hauskatze>	 horizon?
[22:33:58] <Hauskatze>	 geez we have configs in so many places
[22:34:30] <mutante>	 already checked. project puppet Hiera section is empty, prefix has no prefixes
[22:34:55] <mutante>	 also did Ctrl + F for deployment-tin across that in browser
[22:35:38] <mutante>	 and i just created the instance and applied the role today
[22:35:54] <mutante>	 so it doesnt have old local data
[22:36:03] <Hauskatze>	 mutante: I see plently at https://wikitech.wikimedia.org/w/index.php?search=deployment-tin&title=Special%3ASearch&profile=advanced&fulltext=1&advancedSearch-current=%7B%22namespaces%22%3A%5B666%5D%7D&ns666=1
[22:36:10] <Hauskatze>	 not sure if they're related though
[22:36:57] <mutante>	 well.. i would guess these would all have broken puppet runs then because we agree that deployment-tin has been replaced , right
[22:37:18] <mutante>	 also i can't resolve that name if it hasn't
[22:40:12] <mutante>	 paladox also pointed out earlier that there is deployment_server and scap::deployment_server .. but i set both
[22:40:44] <Hauskatze>	 mutante: where does https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep come from? Horizon?
[22:41:25] <mutante>	 no, i think that is just another optional place to put it and it gets merged into one
[22:41:37] <mutante>	 i am not inside deployment-prep though, fwiw
[22:41:57] <Hauskatze>	 :S
[22:42:01] <mutante>	   RAN: /usr/bin/git clone --jobs 1 http://deployment-tin.deployment-prep.eqiad.wmflabs/scholarships/scholarships/.git /srv/deployment/scholarships/scholarships-cache/cache
[22:42:06] <mutante>	 this is an example
[22:42:15] <mutante>	 it builds this command line
[22:42:18] <mutante>	 from somewhere
[22:42:42] <Hauskatze>	 on -releng they say it comes from openstackmanager probably
[22:43:15] <mutante>	 it comes from /usr/bin/scap deploy-local --repo scholarships/scholarships
[23:01:13] <mutante>	 !log deployment-prep deployment-deploy01 edited /srv/deployment/scholarships/scholarships/.git - replaced deployment-tin with deployment-deploy1 to fix scap / cloning of scholarships app
[23:01:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[23:05:40] <gtirloni>	 !log tools Published stretch-tools and stretch-toolsbeta aptly repositories individually on tools-services-01
[23:05:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:45:13] <mutante>	 !log deployment-prep deployment-deploy01 edited /srv/deployment/iegreview/iegreview/.git/DEPLOY_HEAD - - replaced deployment-tin with deployment-deploy1 to fix scap cloning / puppet 
[23:45:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[23:54:44] <nuria>	 just wondering if https://en.wikipedia.beta.wmflabs.org/ being down is part of planned outage 
[23:54:53] <nuria>	 cc kostajh 
[23:55:03] <nuria>	 does mutante know
[23:58:55] <bd808>	 nuria: not planned. there is a bug that has cropped up after the migration -- T210030
[23:58:56] <stashbot>	 T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030