[02:10:01] bd808: do you know when repository creation might work again in striker / when you'll do a deploy for it? I'm currently using gitlab as a temporary host :p [02:10:43] legoktm: I need to package the deploy. the main blocker to that is me untangling the schema upgrade that needs to be done for another patch I merged today [02:10:51] I'll try to get it out next week [02:11:28] bd808: ok, so you're not stopping for the freeze? [02:11:32] django's orm is nice when you can let it manage the db, but a bit ugly when you are manually patching things [02:11:57] I don't see how breaking striker could effect fundraising :) [02:12:01] :D [02:12:57] I merged you patch first, so I could actually deploy that tomorrow without the other change that needs schema updates [02:13:33] I'll see if I can do that... time is not my friend these days [02:29:13] bd808: no rush, I've just started planning out a few tools I want to create [02:29:49] still not sure what I'm going to name them! [02:30:17] bd808: how do you get a free mifi? :) [02:55:50] !log tools.Zppixbot Deploying [[Gerrit:Ibc9bf9d37965cf29568223215e864dc58e3c]] T182806 (GCI Task deploy) [02:55:50] Zppix: Unknown project "tools.Zppixbot" [02:55:50] T182806: Update About.php for ZppixBot - https://phabricator.wikimedia.org/T182806 [02:56:08] Really stashbot... you gotta be picky? [02:56:27] !log tools.zppixbot Deploying [[Gerrit:Ibc9bf9d37965cf29568223215e864dc58e3c]] T182806 (GCI Task deploy) [02:56:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [02:56:36] smh [03:05:28] :) project names are really case-sensitive? or just the bot [03:05:50] could you have 2 different ones, zppixbot and Zppixbot [03:09:32] Im pretty sure the projects are insenstive [03:11:57] ok :) well then it's a stashbot bug, yea [03:13:36] I may be wrong i dont usually use my shift key [03:13:42] Or my brain sometimes [03:23:45] Project names are lowercase only. They are Unix shell names [03:24:53] mutante: my “free” mifi is a donor gift from calyx [03:25:34] They run tor nodes and do opsec outreach [03:26:35] bd808: oh! cool. i was given a hint at office to ask about it:) [03:26:41] will check it out [03:27:14] There was some legal rider on part of the 4g spectrum that Sprint owns now that requires it to be resold to nonprofits very cheap/free [03:28:12] Calyx has a deal where if you donate $500 or more you get a mifi with unlimited data (US only) [03:28:48] ah,, 500 or more:) it starts to make sense [03:28:50] The next year you only have to give $400. [03:29:06] interesting [03:29:08] thanks! [03:29:28] https://en.wikipedia.org/wiki/Calyx ? [03:29:30] So not free, but actually reasonable for unlimited data. And the money buys TOR nodes [03:29:53] https://www.calyxinstitute.org/member/4G-LTE-FAQ right? [03:30:30] Yeah [03:31:33] i see it's Sprint.. for some reason i associated "mifi" with Verizon but that's just because wmf [03:31:55] having both should be good coverage [03:33:38] Yeah. I honestly haven’t used it a whole lot. It has come in handy a few times when my cable modem has gone down [03:34:24] I had it hooked up to my home router for a while as a backup channel. [03:34:55] But that router died and I haven’t set it up with the new one [03:36:10] in Santa Cruz i have the issue that only the Verizon network seems reliable [03:36:22] i switched my phone to Google Fi.. it seemed such a good idea to have multiple networks [03:36:33] but if they are T-Mobile and Sprint and you are in the wrong location [03:36:39] they are both kind of bad [03:37:25] i think it's because it's right on the coast and nobody wants to setup infra for the ocean [03:38:55] thanks, gotta run (still at the office for once, heh) [03:39:20] o/ [13:52:13] !log tools running 'sudo puppet agent -t -v' in tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs since didn't update in the last run with clush [13:52:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:54:18] !log tools same in tools-exec-1415.tools.eqiad.wmflabs [13:54:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:55:35] !log tools same in tools-checker-02.tools.eqiad.wmflabs [13:55:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:55:46] chasemp: not sure why that happened ^^^ [13:56:27] perhaps the warnings I mentioned? [13:56:37] everything I saw should be informational afaiu [13:57:39] chasemp: Hey! Any means by which we can make some progress on this: https://phabricator.wikimedia.org/T171417? :) [13:58:29] d3r1ck: no idea, if you make a comment on the task I will put it in our weekly meeting talk about [13:59:26] chasemp: Not sure I really understand but let me comment on the ticket now [14:01:31] Left a comment on the ticket! [14:01:59] tx d3r1ck I'll see if anyone knows during our meeting next tue [14:02:30] Okay! Thank you very much chasemp :) [18:45:02] On Toolforge I have a relatively new tool account (created two days ago) that does not have access to tool databases (with both "sql local" or lengthy "mysql ..." as described in wikitechwiki). However, it can connect to replicas and there is a valid replica.my.cnf in the tool's $HOME. My other tool accounts can connect to local toolforge databases, so what's on here? [18:46:27] it says: "ERROR 1045 (28000): Access denied for user 'sXXXXX'@'10.68.23.58' (using password: YES)" (with sXXXXX indicating my mysql user name from the replica.my.cnf) [19:16:34] !help anyone here, and reading? [19:16:34] MisterSynergy: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [19:17:09] hey MisterSynergy [19:17:14] hey bd808 [19:17:21] That sounds like a problem that we have seen before [19:17:29] good to know [19:17:31] i'll look [19:17:34] can I fix it? [19:17:37] thanks madhuvishy [19:18:16] MisterSynergy: it happens on the server side. the script that makes new user accounts sometimes gets "stuck" and needs to be moved along [19:19:03] okay thanks, I'll wait for the repair on your side [19:37:07] jynus: I'm confused https://phabricator.wikimedia.org/T182948#3840439 [19:48:36] Is it worries about wire line copies? Useless with my implementation since user information isn't sent. [20:28:17] MisterSynergy: try now? [20:29:20] thanks, works now! [20:36:13] Hi, just one question regarding the database: it seems, that my databases were moved (deleted?), who can help me in this case? :( [20:38:36] FNDE: Have you seen https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown ? [20:39:02] bd808: Where can I find what images you offer on the cloud? Can't seem to find it on WIkitech [20:39:04] My guess would be that you had application databases on labsdb1003 (c3.labsdb) [20:39:20] Debian? Centos? Ubuntu? etc. [20:39:33] multichill: Debian Jessie and Debian Stretch. Ubuntu Trusty by special request only [20:39:53] So only Debian? No Centos? [20:40:15] no RPM based distros, no [20:40:25] ok [20:40:32] we use the same puppet tree and apt repos as prod [20:41:28] We call it a cloud, shouldn't really matter what OS someone runs on top of it ;-) [20:42:30] multichill: our OpenStack setup is currently designed to hide a whole lot of things that would need to be managed by the user for bring-your-own images to work. [20:42:50] It's something that we are considering for the future, but that is over a year away even if we decide its a good idea [20:43:39] Not sure if that's worth the effort [20:43:46] +bd808 thanks! but I think, I don't understand it right: is not possible in the future to have user created tables..? [20:46:22] FNDE: user created tables can exist, on toolsdb [20:47:03] FNDE: https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Database#User_databases [20:50:49] multichill: there are pros and cons. We have had a very very small number of requests for it though so it seems unlikely unless we find some really cool new thing that it would allow [20:51:22] thx jynus! [20:52:19] bd808: im ignorant so what is rpm? [20:53:19] Is this choice documented somewhere? Couldn't find it on Wikitech [20:53:20] Zppix: the software packaging format formerly known as Red Hat Package Manager -- https://en.wikipedia.org/wiki/Rpm_(software) [20:53:26] Ah [20:53:33] Ty [20:53:50] jynus: To what cluster will the db aliases like nlwiki.labsdb resolve in the end? [20:54:03] multichill: it is probably not documented. it shows up in the UI on horizon.wikimedia.org when you try to create an instance [20:54:45] *.labsdb points to the *.web.db.svc.eqiad.wmflabs cluster today [20:55:06] there is an open question of it we should point it to *.analytics instead [20:55:19] *if we [20:56:31] I would say analytics. That means only people who care switch to the fast cluster [20:56:58] And people like me who have a bunch of slow queries all over the place just end up on the batch cluster :-) [20:57:10] multichill: yeah :) [20:57:27] I'm going to switch the `sql` wrapper to default to analytics soon too [20:57:35] just haven't made time for the code change yet [20:58:06] it's quarterly planning time! so I'm busy writing up what we will be working on next [20:58:24] I noticed some jobs of my timing out so switch those over to the analytics db's. [20:59:03] The distributed cloud with worker nodes in US, EU and Asia? :P [20:59:19] I don't thins we actually have a query killer running for either cluster yet... [20:59:26] *think [20:59:57] there have been some reports of connection drops but we haven't been able to trace them to a concrete cause yet [21:00:03] With nice availability zones? [21:00:14] multichill: a nice dream :) [21:00:29] My timeout issue was because of a OOM. [21:00:47] So nothing with the db [21:01:40] We may have a second AZ after we get the neutron migration done. We have some hardware budget to build a small second end-user facing segment. [21:02:08] It will probably be in the eqiad datacenter still [21:02:14] Oh, Neutron, so ipv6 is incoming? :-) [21:02:38] the list of things neutron will let us start working on is very long :) [21:02:42] bd808: i know its documented but is there a policy on bots and irc logs? [21:03:02] but yes, I hope ipv6 is one of the earlier ones [21:03:02] Toolforge* [21:03:31] Zppix: "don't log channels that say they are not logged in the topic" would probably be the only real policy there [21:03:46] not written but socially understood [21:04:00] Okay just making sure as i was unaware until today that my bot was logging [21:04:29] if you aren't using the logs, turn them off and save the NFS bandwidth :) [21:04:56] the fewer disk reads/writes the better for the shared resource [21:05:17] bd808: agreed ill see if i can do that without breaking the meetbot portion of my bot [21:05:37] Zppix: *nod* [21:05:50] If not ill find some way to hard linit file sizes [21:05:55] Limit* [21:06:06] Company I work for offers cloud services through https://clouds.geant.org/geant-cloud-catalogue/geant-cloud-catalogue-iaas/ and quite a lot is ipv6 these days :-) [21:06:50] In my ideal universe we'd quickly get to a place where ipv6 is default and you have to ask for ipv4 :) [21:07:25] That's called Facebook [21:07:53] or the place the internet was supposed to have gotten to 15 years ago [21:08:19] I dont even think prod has ipv6 [21:08:30] it does [21:08:37] I access most things over ipv6 [21:08:47] Must not be public facing ipv6 then [21:08:59] Because im always sent through ipv4 [21:09:25] $ dig AAAA +short en.wikipedia.org -- 2620:0:861:ed1a::1 [21:09:40] Finally have an ISP with native v6 after using a tunnel for over 10 years [21:09:58] multichill: nice! I'm still using he.net's tunnels [21:10:00] Weird my router is set to favor ipv6 [21:10:24] Probably dns stuff [21:10:47] * paladox mine uses ipv6 too by default. It's switched on for all bt smart hub's. [21:10:52] Zppix: if available it's usually always preferred [21:10:54] I used https://www.sixxs.net/main/ [21:11:15] with dual-stack it will just fall back to v4 if v6 doesnt work [21:11:21] I started with sixxs but switched to he.net probably 6-7 years ago [21:11:32] remember that HE certificate you could get ?:) [21:11:39] mutante: hmm maybe im just unlucky [21:11:54] multichill: heh. I used to have it on my vanity site homepage [21:11:55] We do get full ipv6 routing from HE, that's quite nice [21:13:01] The tunnel node was actually in the DC I worked in so I recall power cycling it when I lost my connection ;-) [21:13:24] Zppix: it's mostly about the work of adding proper AAAA records for everything in the DNS repo and then along with it add these puppet code lines in ops/puppet to nodes: interface::add_ip6_mapped { 'main': } [21:13:33] (but also gotta adjust ferm rules) [21:13:48] a bunch of prod things have it.. but not all [21:14:06] * Zppix pretends to understand puppet [21:15:07] in this case all you need to know is that the term "mapped" means it will add an IPv6 address like this: example Gerrit2001: [21:15:10] For some enviroments I'm considering switching to ipv6 only so I don't have to do everything twice [21:15:16] 208.80.153.106 [21:15:22] See if i have to do puppet stuff i usually just ask paladox to do it lol i tried learning it and i couldnt even get passed the first page lol [21:15:24] 2620:0:860:4:208:80:153:106 [21:15:34] see how the v4 address shows up inside the v6 address there? [21:15:38] multichill +1 [21:15:39] that's what it means by "mapped" [21:15:54] mutante: That looks like the SURFnet convenion..... [21:15:56] see https://phabricator.wikimedia.org/T142807#3841790 [21:15:57] *convention [21:16:09] multichill: i think puppet is ruby? [21:16:12] Iirc [21:16:29] We do only the last octet the same for ipv4/v6 to avoid confusion [21:16:31] multichill: In my ideologically ideal world Cloud VPS would become all IPv6 with a 4in6 gateway that can bridge to the legacy IPv4 Internet [21:16:48] bd808: and unlimited storage and bandwith [21:17:06] sure. and ponies/unicorns for everyone ;) [21:17:12] And goats [21:17:22] I'll take 2 of each of those please [21:17:26] Lol [21:17:33] Exactly. Last time I checked I think SNMP was one of the parts not working properly [21:17:40] jynus: ok. I'll patch the DNS. should be an easy switch [21:18:11] Id be interested in the routing back end of all this (im actually in process of getting a ccna) [21:18:48] Its got to be nuts [21:19:39] bd808: i think i found a way to hard limit my log size [21:20:10] Zppix: What is nuts about it? Ipv4 and ipv6 routing is basically the same, just different address family [21:20:23] Ill try to keep it 24 hrs worth of log and auto delete every 25 hrs [21:20:54] multichill: not what i meant [21:20:56] bd808: is it true, that I can not join user created tables on the new server with the wikipedia tables? :/ [21:21:39] Toolforge cron is it still just crontab -e or is that k8s now too? [21:21:48] FNDE: yes [21:22:05] Zppix: crontab -e is still grid [21:22:26] Hmm [21:22:30] and the host that executes cronjobs has no knowledge of k8s iirc [21:22:40] Ok [21:22:59] zhuyifei1999_: yep, cron is still grid only [21:23:00] I just wondering if i should do cron or just a bash script.. [21:23:20] zhuyifei1999_ too bad :( [21:23:31] a cronjob that runs a bash script? [21:24:07] mutante: well grid is kinda being moved to k8s stuff (atleast it seems that way) [21:24:15] Plus all my stuff is on k8s as is [21:24:33] k8s is the future! or something :) [21:24:47] we'd like to see all of the webservices move to k8s [21:24:58] I'm hoping to do some new outreach about that in the coming months [21:25:01] I use k8s for my bot and web :) [21:25:56] I just dont know how feasiable making this simple thing for k8s [21:26:11] what thing? [21:26:17] Auto log cleaner [21:26:41] All it will do is rm /dir/logs/file [21:26:47] sounds do-able with a bash script? [21:26:51] Yeah [21:26:57] sounds like logrotate [21:27:11] mutante: but a lot more basic [21:27:20] and then add that to crontab [21:27:36] you could copy existing logrotate config from prod repo [21:27:51] mutante: will it work on toolforge? [21:28:04] Zppix: that may not work like you hope. If the bot keeps the file handles open then rm will make it look like they are gone but really the data stick around as an orphaned inode on the NFS server [21:28:14] Oh [21:28:16] oh, I see now how invonvinient that change is [21:28:22] *inconvenient [21:28:35] jynus: not a big deal. search and replace :) [21:28:36] bd808: im open for any ideas then [21:28:57] Zppix: what's the code that logs the files? [21:28:57] jynus: the idea is that we won't be updating that file going forward [21:29:02] yeah [21:29:12] hopefuly the division will catch on [21:29:19] zhuyifei1999_: let me go grab it [21:29:32] old tools server told me they failed to get traction on it [21:29:37] *admins [21:29:48] change is hard [21:29:54] bd808: if it gets too bad we can pool 2 servers for analyticvs [21:29:57] zhuyifei1999_: actually its in the base code i cant change it... its hardcoded [21:30:04] in 1:1 shared load [21:30:10] Zppix: can you paste? [21:30:16] that is possible with the server fully in read only now [21:30:29] nice! [21:30:30] so many wins [21:30:32] zhuyifei1999_: yeah let me get this in pastebin [21:31:03] jynus: $ host enwiki.labsdb [21:31:03] enwiki.labsdb is an alias for s1.analytics.db.svc.eqiad.wmflabs. [21:31:03] s1.analytics.db.svc.eqiad.wmflabs has address 10.64.37.14 [21:31:22] things should start switching over as they reconnect [21:32:15] zhuyifei1999_: https://www.irccloud.com/pastebin/xoFiCvrf [21:32:53] I will monitor the load [21:33:02] and where does it access the files? [21:33:22] zhuyifei1999_: as in where are logs stored? [21:33:39] jynus: also do not forget to sleep and live your non-wikimedia life ;) [21:33:58] I mean where does it call open() and/or file.write() [21:35:56] Hmm i cant see where [21:37:42] I think is what ill do is ill do a bi-weekly maintaince downtime and kill the bot and manually do rm cmd (it probably needs to be done anyway [22:17:24] he left, but with lxc/docker/etc. it's pretty easy to run whatever OS you really want in cloud vps [22:19:50] true. not as the base image, but in a vm in a vm setup [22:20:00] its vms all the way down folks! [22:20:25] bd808: also, I'm not sure if you all are having a weekly meeting next week but I was hoping https://phabricator.wikimedia.org/T182946 could get approved before everyone left so I can play with it over the break [22:20:36] I read some blog post about kubernetes running on open stack running on kubernetes [22:21:06] oh nice! [22:21:20] we should be having our regular meeting next week [22:21:54] Horzions logo bugs me but im not artistic enough to fix it [22:22:05] I think I looked at hound at some point a couple of years ago, but then decided that github org search was good enough [22:22:38] Zppix: jsut that it is a bit blurry? [22:22:44] * bd808 squints [22:23:14] I've been using codesearch.debian.net recently and was looking to see how reusable it was (it really isn't) but found hound and a few other things that looked promising [22:23:18] That and the name i dont know of this "Labs" thing is, sounds creepy xD [22:23:46] Zppix: I'm supposed to fix that... let me find the file in Puppet :) [22:24:01] Oh so we have a new image? [22:25:02] the "unicorn rampant on a field of green within the Wikimedia Community border" is the Cloud Services / Cloud VPS logo [22:25:11] but I need to fix the text [22:25:25] and I can probably make it less blurry at the same time [22:25:54] Use comic sans xd [22:26:26] Woah my irc client made comic sans comic sans creepy much [22:33:20] hehhe, google "the most hated font" without ever mentioning the name [22:40:00] * zhuyifei1999_ never understand why comic sans is hated [22:44:11] https://www.huffingtonpost.com/2013/08/29/vincent-connare_n_3837441.html [22:45:22] "Connare was working at Microsoft in 1993, when he was asked for his input on a new program’s fonts. “When I loaded the CD a little dog came up. He talked in a speech balloon like you would get in a newspaper cartoon strip, but it was in the system font Times New Roman,” Connarre says. “I thought, ‘That’s silly. Dogs don’t talk like that.’ So I said it would look better if it [22:45:28] looked like a comic book.”