[00:21:28] <awight>	 !log deployment-prep ORES with git-lfs
[00:21:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[00:33:15] <paladox>	 awight: I think you need to set
[00:33:20] <paladox>	 git_binary_manager: git-lfs 
[00:33:29] <paladox>	 In scap.cfg
[00:35:13] <awight>	 oho, paladox thank you!
[00:35:51] <paladox>	 Your welcome :)
[00:35:59] <paladox>	 awight: did that work? :)
[00:36:08] <Amir1>	 !log wikilabels u_wikilabels=> update campaign set active = 'f' where wiki = 'fawiki' and id != 71;
[00:36:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[15:47:28] <awight>	 !log deployment-prep ORES with git-lfs, scap config
[15:47:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[15:48:38] <zhuyifei1999_>	 halfak: I’ll deploy that quarry thing next hour hopefully. Outside right now. Thanks for reviewing the patch
[15:56:45] <zhuyifei1999_>	 !log quarry deploying d653400 to quarry-runner-0{1,2} T188564
[15:56:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[15:56:47] <stashbot>	 T188564: Quarry should refuse to save results that are way too large - https://phabricator.wikimedia.org/T188564
[15:58:44] <zhuyifei1999_>	 halfak: can you test if it works now?
[16:07:36] <arturo>	 !log puppet3-diffs add myself to the project as admin so I can update compiler facts
[16:07:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet3-diffs/SAL
[16:07:40] * halfak tries
[16:08:41] <zhuyifei1999_>	 the task get killed without status updating if I kill it manually via SIGALRM
[16:09:28] <halfak>	 zhuyifei1999_, it failed
[16:09:29] <halfak>	 https://quarry.wmflabs.org/query/25502
[16:09:30] <zhuyifei1999_>	 (i.e. the signal handler seems to be the default one if I do it manually)
[16:09:39] <halfak>	 Hmm.  
[16:09:50] <halfak>	 Writing 92k rows with one int should be fast, right? 
[16:09:57] <halfak>	 Do you see a spike in IO? 
[16:10:24] <zhuyifei1999_>	 looking
[16:10:35] * halfak looks back at the code. 
[16:11:31] <halfak>	 Hmm.  Looks right. 
[16:11:36] * halfak runs some tests
[16:11:49] <zhuyifei1999_>	 yeah IO spike right now
[16:12:11] <halfak>	 Oh.  Hmm.  Shouldn't be related to the query that just failed, you'd think
[16:12:28] <zhuyifei1999_>	 (when I ran my forked query)
[16:14:15] <zhuyifei1999_>	 halfak: shall I switch back to the un-limited version and see how long the save takes?
[16:14:28] <halfak>	 Just ran a test on the analytics machines and the save took less than .5 seconds. :| 
[16:14:48] <zhuyifei1999_>	 ik, but it saves the query results to NFS
[16:14:54] <halfak>	 OH!
[16:15:01] <halfak>	 maybe NFS is unhappy :| 
[16:15:16] <zhuyifei1999_>	 which is... one of the slowest things here
[16:15:25] <halfak>	 damn.  
[16:15:33] <halfak>	 Can never get away from NFS
[16:16:14] <halfak>	 FWIW, the output file from my test is 809K -- which seems reasonable. 
[16:16:22] <halfak>	 This is the default TSV format that MySQL outputs
[16:16:33] <zhuyifei1999_>	 if I switch to instance local storage I doubt the storage space of single-instances can handle the amount of query outputs
[16:16:41] <halfak>	 zhuyifei1999_, right. 
[16:16:42] <halfak>	 Hmm
[16:16:53] <halfak>	 I think there are some option for big-disk instances
[16:17:22] <zhuyifei1999_>	 how about let's just switch back to unlimited and see how long it takes?
[16:17:56] <halfak>	 I'm down for attempting that but I need to step away from the computer soon. 
[16:18:06] <halfak>	 Can you test in my absence for the next ~hour?
[16:18:13] <zhuyifei1999_>	 ok
[16:18:21] <zhuyifei1999_>	 !log quarry depool quarry-runner-01
[16:18:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[16:18:53] <halfak>	 o/
[16:20:57] <zhuyifei1999_>	 !log installed python-dbg on quarry-runner-02 because it's so good
[16:20:57] <stashbot>	 zhuyifei1999_: Unknown project "installed"
[16:21:04] <zhuyifei1999_>	 !log quarry installed python-dbg on quarry-runner-02 because it's so good
[16:21:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[16:23:08] <zhuyifei1999_>	 py-bt is empty o.O
[16:23:55] <zhuyifei1999_>	 oh, wrong version of python running...
[16:29:04] <zhuyifei1999_>	 !log quarry quarry-runner-02 is on d9cc1c8
[16:29:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[16:30:28] <wikibugs>	 (03PS1) 10Volans: Puppetboard: add dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/419775
[16:53:42] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 031] Puppetboard: add dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/419775 (owner: 10Volans)
[16:54:14] <wikibugs>	 (03CR) 10Volans: [V: 032 C: 032] Puppetboard: add dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/419775 (owner: 10Volans)
[16:56:33] <zhuyifei1999_>	 !log granted elasticsearch credentials to tools.denkmalbot T185624
[16:56:34] <stashbot>	 zhuyifei1999_: Unknown project "granted"
[16:56:34] <stashbot>	 T185624: Elasticsearch credential request for denkmalbot - https://phabricator.wikimedia.org/T185624
[16:56:40] <zhuyifei1999_>	 !log tools granted elasticsearch credentials to tools.denkmalbot T185624
[16:56:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:10:38] <wikibugs>	 (03PS1) 10Volans: Add keyholder dummy keys for puppetboard [labs/private] - 10https://gerrit.wikimedia.org/r/419796 (https://phabricator.wikimedia.org/T184563)
[17:14:24] <wikibugs>	 (03CR) 10Volans: [V: 032 C: 032] Add keyholder dummy keys for puppetboard [labs/private] - 10https://gerrit.wikimedia.org/r/419796 (https://phabricator.wikimedia.org/T184563) (owner: 10Volans)
[17:34:07] <halfak>	 o/ zhuyifei1999_ 
[17:34:09] <halfak>	 Sorry was AFK for longer than expected. 
[17:34:32] <zhuyifei1999_>	 hi
[17:35:02] <zhuyifei1999_>	 so I tested your query and the whole run seems a lit shorter than 5 mins
[17:35:14] <zhuyifei1999_>	 I guess we can double to 10?
[17:35:49] <halfak>	 Writing out 800K takes 5 mins?
[17:35:57] <halfak>	 Or the query + the write time?
[17:36:09] <zhuyifei1999_>	 the query finish is seconds
[17:36:11] <zhuyifei1999_>	 *in
[17:36:27] <zhuyifei1999_>	 so the query+write
[17:37:00] <zhuyifei1999_>	 (if the explain says nothing then it is probably writing)
[17:38:30] <halfak>	 Strange that it would take so long.  It seems like this is an issue for the NFS-based setup.  
[17:38:42] <halfak>	 zhuyifei1999_, do you have a good way to check how much storage quarry is currently using?
[17:38:55] <zhuyifei1999_>	 du? proabably
[17:38:57] <zhuyifei1999_>	 https://phabricator.wikimedia.org/T178520 fyi
[17:39:15] <chicocvenancio>	 Graphite? 
[17:39:29] <halfak>	 Oh nice.  
[17:39:38] <halfak>	 OK.  So I think the new big-disk instances are ~300GB
[17:39:48] <halfak>	 andrewbogott, ^ 
[17:39:51] <halfak>	 Can you confirm?
[17:40:36] <zhuyifei1999_>	 the other problem is how to send the query results to the bigdisk, whereever it is
[17:40:39] <andrewbogott>	 halfak: that's right
[17:41:42] <zhuyifei1999_>	 and what if the instance is gone... query results is a snapshot of a specific state and is not rebuildable
[17:43:15] <zhuyifei1999_>	 another way would be kill on memory usage > a value, but that would be complicated
[17:43:55] <zhuyifei1999_>	 neither the malloc-ed size and resident size is accurate on how much memory is 'used'
[17:46:06] <zhuyifei1999_>	 (they accumulate asymptotically for a non-memory-leaking program)
[17:48:13] <halfak>	 zhuyifei1999_, maybe we write primarily to the bigdisk and do a periodic backup to nfs or a second bigdisk?
[17:48:27] <zhuyifei1999_>	 hmm
[17:48:39] <zhuyifei1999_>	 yeah sounds good to me
[17:48:49] <halfak>	 andrewbogott, sorry to bug again.  Any better option for a long-term backup of impossible to reproduce data in cloud VPS?
[17:49:11] <zhuyifei1999_>	 so how to send the query results to bigdisk?
[17:51:12] <zhuyifei1999_>	 (when I think of virtual file IO I keep on thinking of FUSE, then I realize nobody here actually run fuse according to puppet, so I'd have to be very adventurous to go that way)
[17:51:45] <andrewbogott>	 halfak: not really.  I think if you search phab you'll find tickets about pending backup systems.
[17:56:09] <chasemp>	 yes in theory we have something on the horizon (pun!) but nothing super near-term, procurement and then time to implement
[17:56:19] <chasemp>	 halfak: out of curiosity how big are we talking storage wise?
[17:57:04] <zhuyifei1999_>	 zhuyifei1999@quarry-main-01:/data/project/quarry$ du -sh results
[17:57:04] <zhuyifei1999_>	 127G	results
[17:57:31] <chasemp>	 ack
[17:57:32] <zhuyifei1999_>	 112G in Oct 2017 https://phabricator.wikimedia.org/T178520
[17:57:46] <chasemp>	 zhuyifei1999_: would it compress well?
[17:58:24] <zhuyifei1999_>	 I can try, but unlikely. It's sqlite databases
[17:58:50] <chasemp>	 usually to backup a sqlite database I have dumped to text and compressed
[18:00:10] <chasemp>	 http://www.sqlitetutorial.net/sqlite-dump/
[18:02:15] <zhuyifei1999_>	 zhuyifei1999@quarry-main-01:/data/project/quarry/results/1232/18542$ gzip -vc 176005.sqlite > /dev/null
[18:02:15] <zhuyifei1999_>	 176005.sqlite:	 51.4%
[18:02:22] <zhuyifei1999_>	 random selection ^
[18:02:30] <chasemp>	 not bad
[18:03:41] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/wq6q9UwH/
[18:03:45] <zhuyifei1999_>	 a few more
[18:04:00] <zhuyifei1999_>	 looks like smaller databases compress worse
[18:04:31] <chasemp>	 less means less to dedupe in the case of text usually
[19:27:29] <zhuyifei1999_>	 !log quarry switch back to d9cc1c8 on both hosts
[19:27:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[20:45:45] <zhuyifei1999_>	 halfak: any suggestions on how to send the results from the query runners to the bigdisk?
[21:35:50] <MaxSem>	 hey, how can we go about deploying https://gerrit.wikimedia.org/r/#/c/403833/ ? we've finally received a DBA approval for the underlying table, but we need a Sanitarium config for it
[21:53:01] <Reedy>	 MaxSem: Needs manual rebase
[21:58:47] <Reedy>	 MaxSem: You might be able to ask bstorm_ nicely as she's been working on the script
[22:01:12] <bstorm_>	 To deploy that, it needs a rebase, merge and then puppet-merge, but the sanitarium config is more the DBAs I think.  
[22:01:27] <halfak>	 zhuyifei1999_, I saw your ping here and added some thoughts in the task :) 
[22:01:36] <zhuyifei1999_>	 k
[22:01:50] <zhuyifei1999_>	 I responded as well
[22:02:58] <bstorm_>	 MaxSem: Do you need sanitarium stuff?  The comment from Jaime seems to suggest otherwise?
[22:03:24] <bstorm_>	 I can get that in there up to the sanitarium  
[22:05:30] <MaxSem>	 rebased
[22:05:49] <MaxSem>	 he seems to be unsure it this patch has already been merged or not
[22:05:57] <J-Mo>	 question for bd808, @here: hi everyone. Looking for some info. I have a tool (HostBot) that's been offline since August, and the tool has (had?) a public user db on the old replica servers. I'm just now seeing this blog post https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_servers_ready_for_use/  has my old db deleted, or migrated to a new server?
[22:06:20] <bstorm_>	 MaxSem: I'll check :)
[22:06:59] <Platonides>	 J-Mo: I suspect it would be gone
[22:07:18] <J-Mo>	 mmm. was afraid of that. thanks Platonides
[22:07:26] * J-Mo goes hunting for his latest backup
[22:07:33] <Platonides>	 I'm not completely sure about that
[22:08:06] <Platonides>	 but I think there were some warnings about not keeping some dbs
[22:08:23] <MaxSem>	 so my question is shall I create the table now or the puppet change needs to be first
[22:08:50] <bstorm_>	 The puppet change is just for creating the labs views after the fact.
[22:09:09] <zhuyifei1999_>	 J-Mo: iirc there was a backup
[22:09:19] <bstorm_>	 If you need to create a table, this won't show up on the Cloud DB views until after we manually run the script anyway MaxSem
[22:09:33] <zhuyifei1999_>	 before the complete shutdown
[22:09:46] * zhuyifei1999_ checks
[22:09:52] <bstorm_>	 The puppet change is so that a view will also get created...once we run the scripts.  The table has to be there before we run it.
[22:10:02] <bstorm_>	 But it doesn't run on its own
[22:10:17] <bstorm_>	 Does that help?
[22:12:07] <zhuyifei1999_>	 J-Mo: https://phabricator.wikimedia.org/T183758#3887502
[22:13:20] <J-Mo>	 thanks zhuyifei1999_ I'll go hunt in scratch
[22:13:42] <zhuyifei1999_>	 np
[22:15:38] <J-Mo>	 yep. found it! what a relief
[22:20:09] <chasemp>	 J-Mo: I think they archived them somewhere during as well
[22:21:38] <J-Mo>	 chasemp thank you! I've got the zipped archive downloaded. Will migrate it to tools.labsdb
[22:25:48] <bstorm_>	 MaxSem: What wiki/db should the table exist on at this point?  I was just poking to see if the table is created yet.  I don't see it on enwiki at least
[22:26:11] <MaxSem>	 ok, thanks bstorm_. I'll create the table and populate it with a couple of test rows
[22:26:47] <bstorm_>	 Ok :)
[22:27:09] <Reedy>	 bstorm_: it'll be on the centralauth db
[22:27:16] <bstorm_>	 Oh!
[22:27:17] <bstorm_>	 Ok
[22:27:24] <Reedy>	 only one, not hundreds :)
[22:29:34] <bstorm_>	 In that case, it isn't in that db either on our replicas
[22:31:31] <bstorm_>	 That means it is either not in prod or not on the sanitarium, etc.
[22:31:50] <bstorm_>	 either way, it's not there yet, so the puppet patch doesn't need to merge yet
[22:32:25] <bstorm_>	 Once it is replicating through, then the puppet patch will be ready, per se.
[22:47:55] <MaxSem>	 bstorm_: the table is up
[22:48:21] <bstorm_>	 I see it on one of the replicas
[22:48:57] <bstorm_>	 So from that point, we should be able to merge this and run the script, which will cause it to show up in the clients
[22:49:13] <bstorm_>	 well to be available to them at least on the replicas
[22:55:00] <bstorm_>	 MaxSem: do you have the ability to merge on puppet or do you need me to push it through?
[22:55:25] <MaxSem>	 no, I'm not an ops =)
[22:55:31] <bstorm_>	 Ok, I'll get it
[22:57:02] <bstorm_>	 Merged
[23:00:31] <bstorm_>	 Now for the table to show up on the wiki replicas, it still needs to be run.  I have another thing I'm working on that this is waiting behind.  
[23:01:01] <bstorm_>	 If I can get that cleared up soon, I'll run the scripts
[23:02:53] <MaxSem>	 thanks bstorm_!
[23:12:16] <wm-bot>	 This user is now online in #wikimedia-cloud. I'll let you know when they show some activity (talk, etc.)
[23:12:16] <zhuyifei1999_>	 @notify halfak what do you think of just increasing the time limit to 10 mins for the time being, before we come up with a plan for moving the data?
[23:13:12] <zhuyifei1999_>	 why don't I see their name here...
[23:18:09] <Reedy>	 The bot lies
[23:18:27] <zhuyifei1999_>	 lol