[10:07:25] arturo: bstorm_ hey, how would you feel if drop around a 1TB from wikidata replica in a month or so? [10:20:30] Amir1: with drop you mean? [10:20:43] arturo: dropping a table [10:20:43] anyway, 1TB over a month should be fine! [10:21:20] Amir1: I suggest a phab task anyway, for future reference as well [10:21:34] https://phabricator.wikimedia.org/T208425 [10:22:17] 👍 [10:22:51] hi, I'm getting (seemingly) random 500s from horizon.wikimedia.org/api when browsing instances and/or trying to create a new one, known? labweb1002 shows up in the headers as the host that's serving requests [10:23:27] ok now got a 500 even reloading the horizon ui [10:23:48] https://horizon.wikimedia.org/project/images/ that is [10:24:23] we might need to restart apache [10:26:30] !log admin restarting apache in both labweb1001/labweb1002 upon reports of returning 500s [10:26:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:26:33] godog: try again [10:28:04] arturo: looks good now, thanks! [10:28:10] \o/ [13:28:03] !log admin disable puppet on labweb100[1,2] to enable horizon event traces T240852 [13:28:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:28:05] T240852: CloudVPS: horizon giving http/500 intermitently - https://phabricator.wikimedia.org/T240852 [14:49:49] Amir1: I think the only help you will need from the Cloud Services team for T208425 will be dropping the view of the wb_terms table once it is empty/unused. Does that sound correct to you? [14:49:50] T208425: [EPIC] Kill the wb_terms table - https://phabricator.wikimedia.org/T208425 [15:29:23] for a couple days in a row, I noticed shinken-wm was showing low free space warnings for cvn-app8 in #countervandalism, which didn't seem normal. Is that something that might be a problem that needs to be fixed? [15:45:50] bd808: yup, also is it possible (given that the table is not being written anymore) to remove it from master and sanitrum hosts while keeping it in labs replica? [15:46:01] as tool builders are using this probably [15:46:52] !log tools.zppixbot-test restarted -test pod to deploy eee8d55 and fix status moodule [15:46:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [15:47:29] Amir1: the replicas are a mirror of the sanitarium hosts which are in turn mirrors of the prod masters. So basically seconds after the masters change the replicas change. We don't "freeze" old data at the replica layer. [15:47:49] and "seconds" is usually actually milliseconds [15:47:59] bd808: noted thanks [15:48:39] bd808: last question, is dropping this table help the replicas you have in matter of storage? [15:49:08] around 1TB uncompressed, 700 GB compressed [15:51:03] Amir1: Less data never hurts :) But the DBA folks are the ones who monitor and support that aspect of the replicas [15:51:47] DBAs are dancing about s8, I was thinking of the cloud replicas but I think I should ask them too :P [15:52:46] the wiki replicas are close to needing a full redesign. Table sizes are getting too big to keep all of the shards on the same mysql/mariadb instance [15:53:47] And it seems hard for me to believe, but all the hardware we are using there is up for replacement in the coming fiscal year. Time flies :) [15:53:48] don't know how MariaDB handles it, specifically, but on most systems I see, disk space isn't reclaimed unless the database is closed and compacted [15:57:15] bd808: https://phabricator.wikimedia.org/phame/post/view/195/coming_to_terms_with_change/ :) [16:00:04] Amir1: do y'all have any sense of how impactful this will be for tools? I know the actor table changes led to a lot of code changes in the tools space, but I have no good sense about wikibase specific things. [16:00:32] we should certainly send something out on the cloud-announce@ list [16:00:46] bd808: it might be, it was already annoucned a year ago [16:00:53] new announcements are coming [16:00:56] nobody will remember that :) [16:12:08] you all mentioning the lists, reminded me I never signed up on them for the cloud lists - got that taken care of now [17:38:44] Has anyone gotten Microsoft SQL Server Tools to connect to the replicas? [17:45:09] Someone asked that a while ago, the response was mostly "Why would you want to do that?" [17:48:56] Betacommand: I feel like there was some really outdated advice about that on wikitech somewhere... [17:49:39] AntiComposite: its a nice GUI interface to SQL thats not command line [17:49:51] It should mostly be the same as connecting any other database tool. There is some info about other tools at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_with... [17:50:32] you would need a local ssh tunnel into Toolforge / Cloud VPS to route the traffic over [17:52:14] Ah think thats where it died last time I tried [17:53:25] Betacommand: yeah, the tunnel is likely to be the trickiest bit for folks working in a GUI environment for their normal workflows [17:53:48] bd808, for future reference, here's jennaconley :) [17:54:00] jennaconley: hello! [17:54:03] Hi! [17:55:41] jennaconley: My IRC direct messages ("PM" in IRC lingo) are blocked to folks who do not have a registered nick, but you can ping me somewhere public like this and I can start a PM with you without you needing to learn how to register you nick right away. [17:56:42] If you figure out that irc is good for you otherwise then I'm sure andrewbogott can help you figure out the arcane NickServ system eventually [17:57:08] https://freenode.net/kb/answer/registration [17:57:48] Betacommand: I've only glanced over it, but https://www.digitalocean.com/community/tutorials/how-to-route-web-traffic-securely-without-a-vpn-using-a-socks-tunnel looks like a reasonable tutorial on setting up ssh tunnels [17:58:58] that one is targeted at integrating with a web browser, but the tunnel basics should be the same no matter what you want to point at the tunnel [18:00:01] many blogs and tutorials to choose from -- https://www.google.com/search?q=ssh+tunnel+windows [21:41:30] !log admin restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes [21:41:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:37:53] !log tools.zppixbot restore abuse.py as it seems to be missing (which is causing logspam) and reboot bot to clear reload error [23:37:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL