[00:15:34] <chicocvenancio>	 RileyH: regarding T188684, do you leave a browser open with your server?
[00:15:35] <stashbot>	 T188684: Actively running servers shut down unexpectedly - https://phabricator.wikimedia.org/T188684
[00:15:56] <RileyH>	 Sometimes yes, sometimes no
[00:16:23] <RileyH>	 The last time it happened, no. Previous times, probably once or twice.
[00:16:58] <chicocvenancio>	 if you do leave than it is something new, otherwise it is a known bug
[00:17:31] <chicocvenancio>	 inactivity is defined as time since last "ping" by the browser to the server
[00:17:58] <RileyH>	 Isn't the whole point of this being able to walk away from your computer or close your browser?
[00:18:20] <RileyH>	 If I am understanding you correctly, this should not happen if you leave the terminals open in your browser.
[00:18:44] <chicocvenancio>	 yes, that is the current limitations we have
[00:19:28] <chicocvenancio>	 ideally both inactive but open could be shut down and active but closed could be kept alive (your use case)
[00:19:57] <RileyH>	 Eh..
[00:20:13] <RileyH>	 That doesn't work for me then
[00:20:19] <RileyH>	 I can't have both servers open
[00:20:38] <RileyH>	 I can only have one open at a time due to OAuth
[00:20:58] <chicocvenancio>	 unless yu.vi or some other jupyterhub developer has already solved this (they might), it will probably need significant developer time to solve this
[00:21:23] <chicocvenancio>	 I use more than one account with chrome's profiles, if that helps
[00:21:42] <RileyH>	 Smart.. Could also just use two browsers
[00:21:58] <chicocvenancio>	 or, you can use toolforge itself for the time being
[00:22:33] <chicocvenancio>	 (I dislike the idea of saying PAWS is limited but it is the reality at this point)
[00:23:10] <RileyH>	 I am awaiting toolforge access 
[00:23:24] <RileyH>	 PAWS would be ideal for me if I could access both bots from my main account
[00:24:08] <RileyH>	 Do you have the phab for the known issue?
[00:25:23] <chicocvenancio>	 RileyH: bd80.8 has already aproved your toolforge access
[00:25:50] <chicocvenancio>	 you're [[User:Huntley]], aren't you?
[00:25:56] <RileyH>	 Sweet, that would have been within the last two hours then
[00:25:57] <RileyH>	 Indeed
[00:26:03] <RileyH>	 https://toolsadmin.wikimedia.org/tools/membership/status/255 prerfect
[00:31:00] <chicocvenancio>	 RileyH: looking at phab tasks, I think it as known issue upstream only. I'll turn T188684 into an upstream tracking task once I find the relevant github issue again
[00:31:00] <stashbot>	 T188684: Actively running servers shut down unexpectedly - https://phabricator.wikimedia.org/T188684
[00:33:39] <RileyH>	 Cheers
[01:42:42] <legoktm>	 bd808: do you have a little bit of time to help me understand why codesearch is out of disk space but at the same time not out of space?
[01:42:49] <legoktm>	 or anyone else (!help) :)
[01:42:57] <legoktm>	 df -h shows
[01:42:58] <legoktm>	 /dev/vda3        19G   12G  5.7G  69% /
[01:42:58] <bd808>	 legoktm: sure. what's the instance name?
[01:43:02] <legoktm>	 codesearch2
[01:43:09] <legoktm>	 but I can't even tab complete
[01:43:10] <legoktm>	 legoktm@codesearch2:~$ cd /var-bash: cannot create temp file for here-document: No space left on device
[01:43:21] <legoktm>	 and puppet has presumably been failing too
[01:43:22] <legoktm>	 The last Puppet run was at Thu Mar  1 10:55:00 UTC 2018 (885 minutes ago). 
[01:43:26] <bd808>	 I bet you are out of inodes
[01:43:33] <legoktm>	 https://tools.wmflabs.org/nagf/?project=codesearch shows 6000GB left
[01:43:34] <legoktm>	 oh
[01:43:38] <legoktm>	 (I mean 6GB)
[01:43:42] <legoktm>	 how do I check that again?
[01:44:26] <legoktm>	 df -i
[01:44:27] <legoktm>	 /dev/vda3      1245184 1245184      0  100% /
[01:44:28] <legoktm>	 yep
[01:44:29] <legoktm>	 awesome
[01:44:34] <legoktm>	 thanks bd808 :)
[01:44:45] <Reedy>	 inodes are the worst
[01:44:59] <bd808>	 they are the worst indeed
[01:45:19] <legoktm>	 are inodes monitored by anything?
[01:45:21] <bd808>	 also putting your working data on the / partition is the second worst
[01:45:31] <legoktm>	 where should I be putting data?
[01:46:04] <bd808>	 probably /srv after mounting the extra disk. Is this a small instance or something else?
[01:46:05] <paladox>	 - /srv
[01:46:19] <legoktm>	 lemme check
[01:46:43] <legoktm>	 it's medium
[01:46:52] <legoktm>	 it is in /srv/ right now
[01:46:57] <legoktm>	 but how do I mount the disk?
[01:47:33] <paladox>	 legoktm: a puppet class I think does it
[01:47:34] <bd808>	 there is a puppet role for it ...
[01:47:43] <paladox>	 lsb 
[01:47:56] <legoktm>	 I'm going to create a fresh instance for this
[01:48:08] <bd808>	 yeah, that would be best
[01:48:15] <paladox>	 legoktm: but move your data before applying it otherwise the data will be lost
[01:48:29] <bd808>	 hidden really, not lost
[01:48:32] <bd808>	 I think ...
[01:48:34] <legoktm>	 the data doesn't matter
[01:48:41] <legoktm>	 it's just the index of the git repos
[01:48:45] <bd808>	 I don't think the puppet role blanks the mount point
[01:49:59] <bd808>	 legoktm: applying role::labs::lvm::srv will format and mount the quota the instance has above the 20G that is used for the / partition
[01:50:32] <legoktm>	 ok but does that have more inodes too?
[01:50:51] <bd808>	 well, yes because nothing else will be in there
[01:51:04] <bd808>	 inodes are per-partition
[01:51:28] <bd808>	 we could rebuild the filesystem on a new mount too to have higher inode density
[01:52:02] <legoktm>	 !log codesearch created codesearch3 instance (jessie, medium flavor)
[01:52:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[01:52:11] <legoktm>	 !log codesearch applied role::labs::lvm::srv role to codesearch3
[01:52:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[01:52:27] <bd808>	 the default is something like 1 inode per 512K or something I think...
[01:53:12] <legoktm>	 /dev/mapper/vd-second--local--disk   21G   44M   19G   1% /srv
[01:53:35] <legoktm>	 /dev/mapper/vd-second--local--disk 1343488    11 1343477    1% /srv (inodes)
[01:53:45] <legoktm>	 ok :D
[01:58:49] <bd808>	 legoktm: you might want to unmount and reformat that /srv partition with `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` to get more inode headspace
[01:59:22] <legoktm>	 bd808: ok, is that the full command I need to run?
[01:59:30] <bd808>	 I .. think so
[01:59:50] <bd808>	 https://wiki.archlinux.org/index.php/ext4#Bytes-per-inode_ratio
[02:00:02] <legoktm>	 mke2fs 1.42.12 (29-Aug-2014)
[02:00:02] <legoktm>	 /dev/mapper/vd-second--local--disk contains a ext4 file system
[02:00:02] <legoktm>	 	last mounted on Fri Mar  2 01:52:40 2018
[02:00:02] <legoktm>	 Proceed anyway? (y,n) y
[02:00:03] <legoktm>	 /dev/mapper/vd-second--local--disk is mounted; will not make a filesystem here!
[02:00:20] <bd808>	 yeah, you have to unmount it first
[02:00:28] <bd808>	 sudo unmount /srv
[02:00:56] <bd808>	 you probably should disable puppet first too so it doesn't fight you...
[02:01:22] <legoktm>	 uh, how do I do that?
[02:01:43] <legoktm>	 puppet agent --disable?
[02:01:55] <bd808>	 sudo puppet agent --disable "legoktm messing with partitions"
[02:02:13] <Krinkle>	 legoktm: just for the record, the other codesearch endpoints do work fine (core, skis, ext, skin+ext), it's just the /search (everything) that's not responding
[02:02:17] <Krinkle>	 nicely isolated :)
[02:02:35] <Krinkle>	 E.g. https://codesearch.wmflabs.org/core/ works
[02:02:55] <legoktm>	 they're probably going to fall over sooner or later
[02:03:53] <legoktm>	 bd808: awesome, it now has 5 million inodes instead of 1 million
[02:04:02] <bd808>	 \o/
[02:04:07] <bd808>	 5 times the fun
[02:04:50] <bd808>	 don't forget to `sudo puppet agent --enable` when you are done with manual stuff
[02:04:56] <legoktm>	 just did that :)
[02:05:24] <legoktm>	 !log codesearch ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t bd808) to quintuple the number of inodes
[02:05:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[02:09:25] <legoktm>	 !log codesearch pointed codesearch.wmflabs.org at codesearch3:3002
[02:09:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[02:09:54] <legoktm>	 ok, looks back to normal now ;)
[02:09:58] <legoktm>	 thanks bd808 :)
[02:10:01] <James_F>	 legoktm: Thank you!
[02:10:12] <bd808>	 you did all the work legoktm :)
[02:10:32] <legoktm>	 I would have never figured out the inode thing on my own
[02:10:44] <legoktm>	 is that monitored by anything? I don't see it on nagf
[02:11:01] <bd808>	 fair enough. I heard "why is the disk full when it's not full" and I figured inodes
[02:11:25] <bd808>	 it probably is in the raw monitoring data feed somewhere
[02:12:16] <legoktm>	 !log codesearch deleted codesearch2 instance
[02:12:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[02:16:12] <legoktm>	 I'll be back in a bit and then document everything I just learned :)
[02:26:27] <bd808>	 legoktm: here's your graph of doom -- https://graphite-labs.wikimedia.org/render/?width=586&height=308&target=codesearch.codesearch2.diskspace.root.inodes_percentfree&from=-90days
[02:33:52] <legoktm>	 bd808: haha, that looks perfect
[02:35:14] <legoktm>	 Krinkle: https://github.com/wikimedia/nagf/issues/19 :)
[03:32:01] <Krinkle>	 legoktm: https://tools.wmflabs.org/nagf/?project=codesearch#h_overview_disk-inodes
[03:32:16] <legoktm>	 ooh, thanks :)
[03:44:57] <wikibugs>	 (03PS1) 10Legoktm: Document how to increase inodes during image setup [labs/codesearch] - 10https://gerrit.wikimedia.org/r/415797
[03:45:10] <legoktm>	 bd808: ^ sanity check? :)
[04:11:37] <bd808>	 good night irccloud
[04:12:56] <wikibugs>	 (03CR) 10BryanDavis: [C: 031] "You might want to add in the puppet agent --disable/--enable too." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/415797 (owner: 10Legoktm)
[04:23:25] <wikibugs>	 (03PS2) 10Legoktm: Document how to increase inodes during image setup [labs/codesearch] - 10https://gerrit.wikimedia.org/r/415797
[04:23:42] <wikibugs>	 (03CR) 10Legoktm: [C: 032] "Done, thanks!" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/415797 (owner: 10Legoktm)
[04:23:57] <wikibugs>	 (03Merged) 10jenkins-bot: Document how to increase inodes during image setup [labs/codesearch] - 10https://gerrit.wikimedia.org/r/415797 (owner: 10Legoktm)
[11:56:33] <Ester>	 anyone can help with this? https://phabricator.wikimedia.org/T188496
[11:56:53] <Ester>	 cannot access the dumps 
[13:41:40] <arturo>	 !log tools doing some testing with puppet classes in tools-package-builder-01 via horizon
[13:41:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:19:20] <Sagan>	 general question, is there already a fixed date, when all server with trusty should be migrated to somethind fiferent?
[17:19:27] <Sagan>	 *different
[17:22:22] <jynus>	 Sagan: I cannot answer your question (not a cloud member), but apparently trusty is EOLed on April 2019
[17:22:33] <jynus>	 https://wiki.ubuntu.com/Releases
[17:35:36] <bd808>	 Sagan: we haven't made the timeline yet, but as jynus says it will be before April 2019
[17:45:40] <Dispenser>	 So my script that normally runs 15-20 minutes weekly, just took 127 minutes :-(
[17:46:41] <chicocvenancio>	 Dispenser: do want help figuring out why? 
[17:47:09] <Dispenser>	 I'm guessing its the begging of the month
[17:48:52] <Dispenser>	 chicocvenancio: The think I'd /REALLY/ like to know is why https://phabricator.wikimedia.org/T181401 became 100-1000x slower in a few weeks on both ToolForge and production
[17:50:09] <chicocvenancio>	 That's a bit beyond my abilities. Could table growth explain it? 
[17:50:52] <jynus>	 Dispenser: you know data is not static, right?
[17:51:30] <jynus>	 specially wikidata- there is huge amounts of edits coming every second
[17:51:51] <jynus>	 and wikidata team has been changing the structure to be more efficient in the last months
[17:52:27] <jynus>	 maybe some queries have too adapt, too- but I don't know the details
[17:53:19] <jynus>	 based on this- https://phabricator.wikimedia.org/T181401#3791003 the query is trying to read over 50 million rows
[17:57:59] <jynus>	 Dispenser: checking the query- your query is not well written
[17:58:52] <jynus>	 change it to SELECT 1 FROM wikidatawiki_p.wb_items_per_site WHERE ips_site_page='Harrington Place, Stellenbosch' AND ips_site_id IN ('frwiki', 'eswiki', ...);
[17:59:11] <jynus>	 and you will be able to get use of the index
[17:59:31] <Dispenser>	 The query was running sub 1-sec (probably 0.01 sec) in November
[18:00:11] <jynus>	 I am telling you now and here, the query is badly written
[18:01:59] <Dispenser>	 jynus: How would I find out if we dropped an index?  Specifically wb_ips_site_page: https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/sql/Wikibase.sql#L28
[18:02:17] <jynus>	 check for changes on the sql, yes
[18:02:21] <jynus>	 that would be the case
[18:02:34] <jynus>	 but I would find strange there was an index for that
[18:02:49] <jynus>	 you can ask, however for a custom index, if you indicate which to add and seems reasonable
[18:03:14] <jynus>	 I would suggest changing the queries is the right way based on it
[18:03:40] <jynus>	 != 'enwiki'; cannot be resolved eficiently
[18:04:05] <Dispenser>	 That's more a HAVING statement...
[18:04:16] <jynus>	 my version up there will return the same results 100x faster
[18:05:47] <Dispenser>	 Why isn't it using the INDEX?
[18:06:31] <Dispenser>	 Seriously what happened to it?
[18:06:43] <Dispenser>	 I can't figure out how to browse gerrit
[18:07:23] <jynus>	 phabricator is easier, it has a source code navigator
[18:09:42] <jynus>	 Dispenser: I have commented on https://phabricator.wikimedia.org/T181401#4018690 why it is slow
[18:10:15] <jynus>	 I cannot give you more tips than that, but feel free to comment there further, maybe someone else can help
[18:12:22] <jynus>	 I didn't write it there, but my version of the query takes 0.1 seconds
[18:12:31] <jynus>	 instead of 17 minutes
[18:14:00] <Dispenser>	 jynus: Is there a was I do SHOW INDEXES FROM wikidatawiki.wb_items_per_site; on ToolForge views?
[18:14:54] <jynus>	 I can show them to you, they should be the same than on production (tables.sql, or the extension)
[18:15:07] <jynus>	 actually
[18:15:22] <jynus>	 I will show you, but they should be on the information_schema_p database
[18:16:19] <jynus>	 actually no, the structure is not there
[18:16:25] <jynus>	 let me copy and paste for you
[18:16:47] <jynus>	 this is from labsdb1009, the current analytics replica
[18:18:04] <jynus>	 https://phabricator.wikimedia.org/P6782
[18:18:15] <jynus>	 so I am not denying the structure can change- it does
[18:18:36] <jynus>	 but there is 0 guarantee that it will be stable
[18:19:01] <jynus>	 basically, the "contract" with wikireplicas is that you get access to the internal stuff
[18:19:23] <jynus>	 but the internal stuff can change (unlike the official api, which is stable)
[18:19:25] <Dispenser>	 Last I looked I can't use force index on views
[18:19:42] <jynus>	 my solution doesn't involve force
[18:20:04] <jynus>	 the problem is the !=
[18:20:10] <jynus>	 that is bad performance
[18:20:55] <jynus>	 it could be that some time ago, non-enwiki items were smaller
[18:21:00] <jynus>	 and that worked
[18:21:20] <jynus>	 but that is the case of queries- as data changes, the query plan changes, too
[18:21:38] <Dispenser>	 Long time ago = 2016 to November 2017
[18:21:45] <jynus>	 for "force"
[18:21:55] <jynus>	 we have special views, like user_revisions
[18:22:16] <jynus>	 we can also create those- special view with different quering strategy
[18:22:31] <jynus>	 as I said, propose a way, and we can help :-)
[18:23:19] <jynus>	 but yes, sadly data changes and structure changes
[18:23:59] <jynus>	 I am trying to help you here reducing 100x your query time
[18:24:12] <Dispenser>	 The documentation https://phabricator.wikimedia.org/diffusion/EWBA/browse/master/repo/sql/Wikibase.sql;10a69d8231e94ff39098fd0a680ebf9a3864ec11$28 lists the `wb_ips_site_page` INDEX, but its gone from production.  What other documentation can I use?
[18:24:55] <jynus>	 let me see
[18:28:31] <jynus>	 so, apparently, that was written
[18:28:42] <jynus>	 but never deployed
[18:28:46] <jynus>	 or removed
[18:30:11] <Dispenser>	 Can we make a tool that dumps the actual schema with indexes?
[18:30:13] <jynus>	 yep, it was dropped https://phabricator.wikimedia.org/T179793
[18:30:35] <jynus>	 Dispenser: yes, in fact, I thought information_schema_p had it
[18:30:48] <jynus>	 I think the problem here is other
[18:31:04] <jynus>	 it was requested to be dropped, but the documentation never updated to reflect this
[18:31:39] <jynus>	 I am going to request to be updated
[18:33:34] <jynus>	 so the databases are good, the documentation wasn't
[18:35:27] <jynus>	 the index seems reasonable to be added to wikireplicas only, I would be ok to ask for it, Dispenser
[18:35:44] <jynus>	 although I am not 100% sure it would fix the query
[18:37:08] <Dispenser>	 jynus: Don't want to mess with the databases.  I'm only upset the documentation I consulted for building the query wasn't up to date.
[18:37:51] <jynus>	 I am too :-)
[18:38:04] <Dispenser>	 AND ips_site_id IN (SELECT dbname FROM meta_p.wiki WHERE dbname != 'eniwki') -- Isn't too complicated to add :-)
[18:38:05] <jynus>	 that is why I reopen T179793
[18:38:06] <stashbot>	 T179793: Consider dropping the "wb_items_per_site.wb_ips_site_page" index - https://phabricator.wikimedia.org/T179793
[18:38:27] <jynus>	 my advice on bug reporting is
[18:38:53] <jynus>	 change how you express things, and people will be more receptive
[18:43:44] <Dispenser>	 jynus: I picked up the wrong project.  Apparently the "dyslexia gets better with age" is a complete fucking lie.  Should've pick a project with less people adept at communicating.
[19:23:36] <Sagan>	 bd808: ok, that is long enough. I first thought it would be this year, that was why I asked
[19:57:03] <bd808>	 Sagan: We will probably start talking to everyone about it in August and try to get most people to migrate by January or Feburary.
[19:57:24] <Sagan>	 still enough time for me :)