[00:19:16] So I am not entire sure if my crontab is doing anything? [00:24:47] I tried configuring to run a bot through crontab and my bot wasn't editing. Then I ran the script manually and it worked just fine. [00:30:01] ...oh [00:30:03] Well there we go [00:32:49] Now that I realized there is an error log, I see this error a lot: [00:33:00] /data/project/projanalysis/bin/python3: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found (required by /data/project/projanalysis/bin/python3) [00:33:40] When I run it in the virtual environment it seems fine, but I guess not from crontab. [00:46:20] harej: whats the crontab entry? [00:52:43] harej: did you build the virtualenv on the same OS (trusty/precise) as the one it's running on? [00:53:35] I use trusty. I don't know for certain the virtualenv was built on trusty. [00:55:13] harej: what is the crontab command? [00:56:09] Betacommand I will look it up when I'm at [00:56:12] My computer [00:57:03] harej: OK, I know I have to use the -cwd command to jsub otherwise stuff gets confused [00:58:08] I was using jstart but then I re-read the docs and changed to jsub a few minutes ago since it seemed more appropriate [00:58:16] These are scripts with defined beginnings and ends [01:00:52] harej: here is an example of what I use: [01:00:54] jsub -mem 600M -N NFCC9 python /data/project/betacommand-dev/svn_copy/sql_nfcc9.py [01:52:36] Betacommand: my crontab is: [01:52:46] https://www.irccloud.com/pastebin/Wp3OUHhi [01:53:07] (it was originally jstart but then i changed it to jsub) [01:56:34] harej: I think your doing it wrong, you shouldnt have your own copy of python, just a virt environment [01:58:58] if i'm not mistaken it's just a symbolic link to the regular python [02:01:14] harej: see https://wikitech.wikimedia.org/wiki/Setting_up_Flask_cgi_app_as_a_tool#Set_up_virtualenv [02:01:32] you would need something similar for your virtualenv [02:04:25] I mean I *have* a virtual environment. Which is how I've been able to install modules. [02:05:11] harej: for some reason when submitting it, the environment is lost [02:05:16] legoktm: poke [02:06:16] harej: if you dont get help here drop an email to the list about the proper way to use virtenv, Its probably something simple, I just dont use it so I am unsure [02:10:41] 6Labs, 10Beta-Cluster, 5Patch-For-Review, 15User-Bd808-Test: Move logs off NFS on beta - https://phabricator.wikimedia.org/T98289#1281434 (10bd808) [03:25:52] Betacommand: hey [03:27:49] legoktm: my crontab is being a jerk :( [03:29:24] harej: I have to go to dinner now, but this is what one of my jobs look like: http://fpaste.org/221349/43148774/raw/ [04:10:43] harej: hey! also do ‘-l release=trusty’? [04:10:56] ...that may make a huge difference! [04:11:06] did that work? [04:11:12] I haven't done it yet [04:11:17] ah :) [04:11:27] by default it runs on precise [04:11:38] Whereas I use trusty. [04:11:50] That is why it kvetches about the software versioning being off. [04:12:15] Also, it's currently complaining about my job names. Is there any way to kill those off so that I can have these new things use those name? [04:12:31] * harej refers to: [Wed May 13 03:55:04 2015] there is a job named 'wpx_load_configuration' already active [04:12:31] what is it complaining [04:12:34] oh [04:12:42] qdel wpx_load_configuration [04:12:46] unless someone *else* is using that name [04:13:02] there [04:14:01] New crontab. Looks good to you? https://www.irccloud.com/pastebin/k0zj49QP [04:15:14] harej: try it! :) [04:15:25] * harej waits another 10 minutes [04:16:15] 6Labs, 10Labs-Infrastructure: Input/Output errors in a /home directory - https://phabricator.wikimedia.org/T47609#1281480 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Closing. [04:33:37] yuvipanda: It works! Thank you! [04:33:49] yw! [04:33:52] we should document that somewhere... [04:34:03] if there’s a section on virtualenvs / using python [06:45:33] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 50.00% of data above the critical threshold [0.0] [07:10:33] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0] [07:37:16] Are there any guides for turning a Mediawiki-vagrant setup into a (reasonably secure) production server? [07:37:35] Depends what you mean by "production" [07:37:42] I used Labs-vagrant to set up an instance but I'm not sure it should be made public as-is. [07:38:36] Coren: The instance is going to be used as a spam honeypot in order to learn from patterns that bots and spammers use. I want it to be spammed, but I'd prefer not having any security loopholes. [07:38:54] Coren: Admin passwords and such, maybe. [07:39:55] There's no particular reason why you couldn't do that as-is. Labs doesn't have very good performance for wikis in the general case though I expect that wouldn't be a concern in your case. [07:40:24] Coren: As is meaning just the out of the box Labs-vagrant instance? [07:40:30] After changing the admin password, of course. [07:47:54] 10Tool-Labs: Clean up huge logs on toollabs - https://phabricator.wikimedia.org/T98652#1281595 (10coren) Actually, unlinking a file via NFS should not be any harder on the server than doing it directly - the real thing to watch out for is to make sure it is not currently opened by any process. Something that /c... [07:49:05] Yeah, though you may well want to use the real database for your db backend. [07:55:16] Coren: labs vagrant uses mysql by default [07:55:21] Also hi :) [08:01:45] yuvipanda: Hi to you too. Wait, isn't it like ungoldy late PDT? [08:02:05] I'm in bed yes [08:02:08] :) [08:02:26] I ended up late in the office again... [08:02:59] I feel like I need to put in 1.5x the time when in office to feel 1x productive [08:03:57] Coren: catchpoint has been mostly happy since the nfs resync finished [08:03:58] So that's good [08:04:16] yuvipanda: I saw; which bodes well for the move to raid10 [08:04:16] Coren: do you have an eta for the rsync backup? [08:04:42] yuvipanda: It's in the second maps project now - last time I had to do an rsync it took about 20h past the maps. [08:04:56] Coren: what happened to labstore1002 switch plans? [08:05:04] Are we going to do raid switch first or? [08:05:38] yuvipanda: Raid switch first - debugging the hardware issue with 1002 is complicated because we're not sure how much/whether the shared shevles are interfering. [08:05:53] Aah ok [08:06:19] I'm working off Mark's game plan atm, and it's pretty sane. [08:06:25] Ok [08:07:14] service manifests have been a success I think - I'll announce them later this week [08:07:37] Nobody had complained about webservice not running in a while [08:07:38] We haven't managed to isolate what exactly the issue with 1002 is beyond "it's not an obvious electrical issue", but digging further means we almost certainly want to disconnect the shelves first. [08:07:47] Right [08:08:45] Besides, switching the live service to a server that passes post about 50% of the time doesn't seem like a good idea to me. :-) [08:09:44] Yeah, service manifests have been a rather good success. Have you played with the not-webservice side yet? [08:10:05] Not yet. I am going to tackle cron next [08:10:10] Crontabs next that is [08:10:11] * Coren nods. [08:10:22] Right Now I'm unifying our webservice code [08:10:50] It is currently a bunch of python / bash scripts in ops/puppet [08:11:04] I'm making that a nice python module in tools - webservice package [08:11:18] I added you as reviewer too if you wanna look :) [08:11:44] This will end up rewriting lighttpd-starter and tomcat-starter too I think [08:12:20] Yeah, I'll give the whole thing a look. [08:13:07] I haven't really started rewriting the lighttpd stuff yet [08:13:24] Mostly trying to get the structure right to begin with [08:13:36] And can move on from there. [08:15:24] And now I shall sleep. Good night [08:20:41] Coren: So after changing the Admin passwd, it's safe to make the wiki publicly accessible and ready to receive spam? [08:20:59] Coren: After all the warnings on several pages, it just seems odd that there's only one step to making it reasonably secure. [08:22:08] polybuildr: Well, it's all about the relative use case. Since this is intended to be a honeypot, I'l working from the presumption that you can just wipe and reset it at need and that none of the data is precious. :-) [08:22:42] polybuildr: Most of the normal hardening steps you'd want to take are to reduce the impact of spammers, after all. :-) [08:23:34] polybuildr: But you'll have to monitor it carefully; you don't have infinite resources and the spammers might as well. [08:26:09] Coren: Thanks! :D [08:26:58] Coren: Also, if they do use their infinite resources, what happens to the instance? The DB cannot make any new inserts and maybe the network IO is all taken up by spammers, right? [08:27:57] polybuildr: More likely, your instance will crumble under the load before that. Mediawiki doesn't scale all the well without a caching infrastructure. [08:28:21] Coren: And vagrant doesn't come out of the box with one, does it? [08:28:54] polybuildr: I don't think it does. IIRC, it's fairly easy to set up a memcached for it (yuvipanda would be able to tell you for sure), but not by default. [08:29:58] Coren: Well, I'm in no hurry to start catching spam. I'll ask him later. :) Thanks again! [08:30:15] polybuildr: But to be honest, if your instance ends up being hammered I'd rather have /it/ crumble than sucessfully hammer on the db. :-) [08:31:01] Coren: Ha, okay. :P It's an m1.small, though, so I suppose it will crumble far before it affects Labs. [08:31:27] No doubt it will. [08:38:13] PROBLEM - Puppet staleness on tools-shadow is CRITICAL 100.00% of data above the critical threshold [43200.0] [09:02:12] Coren: here? [09:02:28] mark: Yep [09:02:32] 10:04:40 yuvipanda: It's in the second maps project now - last time I had to do an rsync it took about 20h past the maps. [09:02:40] seems like it's going to take weeks, rather? [09:03:00] It's almost past the worst - both map projects together have some 20m files [09:03:11] it's at 2.5 TB now [09:03:20] Yes, a LOT of very small files. [09:03:29] i doubt it's gonna get much quicker [09:03:36] When I did the switch to the thin volumes, about 90% of the copy was maps [09:03:55] It took less than a day to do the rest [09:04:05] but how much time will the rest of the maps take? [09:05:48] Hm. It's in the second project, but it /does/ look like it just started. :-( It's got 48 million out of 118m total done [09:06:11] it's gonna take a long time [09:06:15] * Coren thinks. [09:06:22] A tarball might be faster for maps. [09:06:27] why? [09:06:47] Fewer open-read small file-close cycles. [09:06:55] (during copy) [09:07:05] i'm not sure why? [09:07:19] it still needs to open/read/close them all :) [09:07:37] Well, the total will be the same, but building a tarball makes it all unconditionally without a bunch of stat() to make sure it'll have to [09:07:54] And all locally. [09:07:54] it always has to, the destination is empty [09:08:25] Sure, but rsync doesn't know that. It still sends every file's stat() to the receiving end, which then queues the transfer. [09:08:44] so I actually tried something different yesterday [09:08:48] I don't think rsync optimizes for that [09:08:52] i allocated a new 41t thin volume on the destination [09:09:00] and then copied the volume block device over [09:09:03] which is sparse of course [09:09:17] so I compressed the sparse blocks to speed up the transfer for that [09:09:17] Oh? I never tried that. How does that fare? [09:09:25] it went at about 30 MB/s on idle bandwidth [09:09:28] on average [09:09:34] and it was writing it sparse to the thin volume [09:09:47] ionice -c idle pv -eprab /dev/mapper/store-backup | pigz -cf - | pv -q -L 50m | ssh -o Compression=no -o CompressionLevel=0 root@labstore2001.codfw.wmnet 'unpigz -c | dd of=/dev/mapper/store-ls1001_now conv=sparse' [09:09:48] Ooo. That's the part I wouldn't have expected. That's very nice actuallyy. [09:09:49] is what I used [09:10:03] it was still gonna take 350 hours [09:10:09] despite being a fair bit faster than the current rsync [09:10:29] Hm. 350 hours would be /loger/ than the current rsync, wouldn't it? [09:10:35] i doubt it [09:10:42] It's got 48m of 118m files done in... 70 hours or so [09:10:49] that's why I'm saying that I think your rsync estimates is wrong ;) [09:11:22] You're correct that it's wrong - I was expecting alphabetical directoies in project/maps so I thought that part was almost done. [09:11:25] it was less impacting on labs nfs actually [09:11:33] because it's more sequential reads [09:11:46] but some drawbacks, very hard to restart that process of course [09:11:50] and i'm not 100% sure it will work [09:11:53] can only tell at the end really [09:12:01] so rather than wait a week and then find that it failed... I canceled it again [09:12:15] Yeah - the sequential reads will be easier on the server for sure. I think it'll tranfer a lot more data though. [09:12:28] it shouldn't much [09:12:33] Those filesystems have a lot of files shorter than 1 block in length. [09:12:34] just because there's no discard/fstrim there's some bloat [09:13:57] Not even counting the unused blocks - if you copy at block resolution you're going to increase the size of a lot of files to the next block size. On average half a block per file. [09:15:08] mark: We can try your technique with a different filesystem. [09:15:25] mark: create a small filesystem, put a couple 100s files in it, and try that. [09:15:26] so after fstrim there are no unused blocks in theory [09:15:28] but didn't want to run that [09:15:29] yeah [09:15:44] but anyway, you were not around, decided to stop it [09:16:30] also reading from the old project volume [09:16:36] goes at about 700 MB/s [09:16:41] because no client load on those disks [09:17:15] Yeah, not likely to be that fast on the live one. [09:17:32] no, caps out at about 30 MB/s before it's starting to get impacting [09:18:23] http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&m=cpu_report&c=Labs+NFS+cluster+eqiad&h=labstore1001.eqiad.wmnet&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS [09:18:39] But right now the copy is 48/118 after 103 hours (just checked) [09:18:49] https://tools.wmflabs.org/meta/ seems to need a restart? [09:18:53] i paused that rsync while I was playing yesterday [09:18:59] (STOP/CONT) [09:19:17] 48 what? [09:19:18] mark: do you know how long you stopped it, rought? [09:19:22] a few hours [09:19:30] 48 million files out of 118 [09:19:37] check the ganglia graph [09:19:47] approx 5 hours I think [09:20:01] cpu load was higher with my metric, but iowait lower [09:20:08] mark: 5h isn't big enough to impact the average all that much. [09:20:22] s/metric/method/ [09:20:29] It looks like rsync keeps about half a million files / hour [09:21:55] = ~220-240h for the whole thing. [09:22:05] let's wait for that then [09:22:11] perhaps we can do the copy to local eqiad using the other method [09:22:16] from codfw [09:22:18] that'll be quick then [09:22:25] wire speed i'm sure [09:22:41] also it will be less fragmented after the rsync [09:22:48] Since codfw is entirely unloaded, I'm pretty sure we can do a sequetial read fast enough to fill wire. [09:22:54] easily [09:23:04] just need 41 TB of thinly allocated storage somewhere [09:23:55] or less actually [09:24:01] we can shrink that volume to closer to actual size for the copy [09:24:07] so we don't need to copy those sparse blocks at all [09:24:12] we can just eliminate them, even easier [09:24:31] gzip minimizes them but it's still not terribly efficient [09:24:49] Well, we can shrink the thin pool to the actual on-disk size [09:24:55] that's what I just said [09:25:06] so yeah let's do that [09:25:37] do any of these boxes have 10G? [09:25:54] Networking? No. [09:28:37] Best we could do is bind two 1g together [09:28:58] wouldn't help for a single stream [09:29:31] If there's no other traffic, true. [09:29:46] and there wouldn't be [09:34:34] 10Tool-Labs: Get rid of toolwatcher, use skeleton homedirs instead - https://phabricator.wikimedia.org/T91235#1281773 (10coren) pam_mkhomedir is only invoked when a logs in for the first time which - by design - can never happen for service users/groups. The user is never actually //created// in the labs projec... [09:38:06] 10Tool-Labs: Grid engine "swallows" quotation marks (double and single quotation marks) and does not recognize pages at cs.wiki (more at "Additional Information") - https://phabricator.wikimedia.org/T74092#1281789 (10coren) @wesalius: Have you tried the console_encoding change yet? [09:48:30] 10Tool-Labs: Cannot start java processes using the grid engine - https://phabricator.wikimedia.org/T69588#1281795 (10coren) The `java/lang/NoClassDefFoundError` exception means that a class that was available at compile time is not available at runtime. If you get the error when on the grid but not interactivel... [09:51:24] 6Labs, 10Labs-Infrastructure: korma.wmflabs.org server not reachable - https://phabricator.wikimedia.org/T98470#1281799 (10Acs) The instance is up now since yesterday. Any clues about the problem? Cheers [09:58:41] 6Labs, 10Labs-Infrastructure, 6operations: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1281802 (10mark) This seems reasonable yes. Let's move ahead with this after the new backups have finished (codfw + eqiad). [10:20:31] PROBLEM - Puppet staleness on tools-exec-catscan is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:22:20] yuvipanda: Are you the one that disabled puppet on tools-exec-canscan? [10:24:15] PROBLEM - Puppet staleness on tools-exec-cyberbot is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:31:52] PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:32:16] PROBLEM - Puppet staleness on tools-exec-gift is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:33:12] PROBLEM - Puppet staleness on tools-exec-wmt is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:33:32] shinken-wm: FWIW, I can't access Wikipedia at all. [10:36:08] PROBLEM - Puppet staleness on tools-exec-08 is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:36:48] PROBLEM - Puppet staleness on tools-exec-14 is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:37:02] PROBLEM - Puppet staleness on tools-exec-07 is CRITICAL 100.00% of data above the critical threshold [43200.0] [10:43:13] .quit [11:34:45] * Coren greatly desires to murder opendj. [12:49:07] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GuZ-MPG was created, changed by GuZ-MPG link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/GuZ-MPG edit summary: Created page with "{{Tools Access Request |Justification=test tools for display of -omics data from wikidata |Completed=false |User Name=GuZ-MPG }}" [12:57:28] Coren: did you see my tool link? [13:04:47] Betacommand: No, was traveling. Give again? [13:06:16] Coren: need to add the filters but http://tools.wmflabs.org/betacommand-dev/cgi-bin/sge_status.py is the basic framework [13:06:40] * Coren looks [13:10:49] Coren: thoughts? [13:11:56] Well, it looks like it works fine, but I can't really gauge whether there's a marked improvement yet. [13:12:10] 6Labs: Reinstall db1009 from zero - https://phabricator.wikimedia.org/T98958#1282071 (10jcrespo) 3NEW a:3jcrespo [13:13:21] Coren: Not sure I would call it an improved version, just a different way of looking at the same data. All of the columns are sortable so you can view/group by different things [13:15:08] 6Labs: Reinstall db1009 from zero - https://phabricator.wikimedia.org/T98958#1282079 (10jcrespo) [13:16:41] Coren: and the memory value sorting actually handles the B/MB/GB correctly, and so does the time sorting for CPU [13:17:14] Yeah, sorty tables made with dumb javascript are not quite as useful. [13:19:21] Coren: had to use css hidden values to get the sorting to work [13:22:16] 6Labs: Reinstall db1009 from zero - https://phabricator.wikimedia.org/T98958#1282099 (10Springle) - add an `m5-master.eqiad.wmnet` CNAME to dns [13:24:16] 6Labs: Get Labs openstack service dbs on a proper db server - https://phabricator.wikimedia.org/T92693#1282100 (10Springle) Note that services should connect to a CNAME, `m5-master.eqiad.wmnet`. Check if we need any special network/vlan rules (think not, but since it is labs-related, ask @faidon) [13:25:22] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282118 (10jcrespo) [13:35:38] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282123 (10jcrespo) [13:43:36] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282124 (10jcrespo) I believe do not have permisions to schedule the host as down in icinga. [13:52:37] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282125 (10jcrespo) [13:52:46] Anyone know how to configure squid? [13:53:32] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 33.33% of data above the critical threshold [0.0] [14:09:14] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282141 (10jcrespo) [14:23:31] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0] [14:35:15] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282196 (10jcrespo) [14:36:05] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282071 (10jcrespo) I now have permissions, Icinga notified downtime scheduled. [14:59:35] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1282237 (10Andrew) Alex, are there any obvious next steps here? Unfortunately pdns doesn't support split horizon, so the alternative to fixing this is ugly, requiring two different dns servers. [15:11:51] 10MediaWiki-extensions-OpenStackManager: cannot create foo.foo.wmflabs.org domain - https://phabricator.wikimedia.org/T56664#1282243 (10Andrew) p:5Triage>3Low [15:14:26] 6Labs: OpenStack manager extension improvements or replacement - https://phabricator.wikimedia.org/T85613#1282251 (10Andrew) 5Open>3Invalid Horizon is under way; I'm not sure this bug is useful anymore. [15:16:17] 6Labs: Add a second pdns/mysql server - https://phabricator.wikimedia.org/T94865#1282257 (10Andrew) p:5Triage>3Normal [15:16:40] 6Labs: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1282267 (10Andrew) p:5Triage>3Normal [15:17:31] 6Labs: Test Ceph for instance storage - https://phabricator.wikimedia.org/T90364#1282273 (10Andrew) p:5Triage>3Normal [15:19:01] 6Labs, 10Labs-Infrastructure: provide bastion redundancy via DNS round robin - https://phabricator.wikimedia.org/T59834#1282277 (10Andrew) p:5Triage>3Normal [15:19:29] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review: Public IPs not being updated from OpenStack Nova plugin - https://phabricator.wikimedia.org/T52620#1282281 (10Andrew) 5Open>3stalled p:5Triage>3Normal [16:07:17] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282460 (10jcrespo) [16:10:03] valhallasw: thanks for adding me to the git reviewers list for tools :) [16:10:25] valhallasw: also look at the webservice patch again? [16:10:32] yuvipanda: will do, later [16:10:51] first grading students' homework [16:10:54] valhallasw: thanks [16:10:56] Haha [16:11:02] also shinken shinken shinken :> [16:53:49] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1203 is CRITICAL 80.00% of data above the critical threshold [0.0] [16:53:53] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL 90.00% of data above the critical threshold [0.0] [16:54:04] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1404 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:30] PROBLEM - Puppet failure on tools-static-01 is CRITICAL 80.00% of data above the critical threshold [0.0] [16:54:32] PROBLEM - Puppet failure on tools-exec-1215 is CRITICAL 80.00% of data above the critical threshold [0.0] [16:54:35] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1402 is CRITICAL 50.00% of data above the critical threshold [0.0] [16:54:40] PROBLEM - Puppet failure on tools-services-02 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:41] PROBLEM - Puppet failure on tools-exec-1213 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:44] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1205 is CRITICAL 50.00% of data above the critical threshold [0.0] [16:54:46] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:47] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 55.56% of data above the critical threshold [0.0] [16:54:51] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:51] PROBLEM - Puppet failure on tools-exec-1203 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:59] PROBLEM - Puppet failure on tools-services-01 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:54:59] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:55:00] those are almost certainly my fault and should clear shortly [16:55:05] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL 80.00% of data above the critical threshold [0.0] [16:55:05] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:55:09] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:55:11] PROBLEM - Puppet failure on tools-submit is CRITICAL 100.00% of data above the critical threshold [0.0] [16:55:40] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:55:41] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1407 is CRITICAL 50.00% of data above the critical threshold [0.0] [16:55:44] PROBLEM - Puppet failure on tools-trusty is CRITICAL 60.00% of data above the critical threshold [0.0] [16:56:06] PROBLEM - Puppet failure on tools-exec-1403 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:09] PROBLEM - Puppet failure on tools-exec-1408 is CRITICAL 66.67% of data above the critical threshold [0.0] [16:56:11] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1208 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:17] PROBLEM - Puppet failure on tools-static-02 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:17] PROBLEM - Puppet failure on tools-exec-1205 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:21] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:33] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:39] PROBLEM - Puppet failure on tools-exec-1402 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:56:53] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:12] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:24] PROBLEM - Puppet failure on tools-redis is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:26] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1201 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:34] PROBLEM - Puppet failure on tools-exec-1410 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:34] PROBLEM - Puppet failure on tools-master is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:42] PROBLEM - Puppet failure on tools-redis-slave is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:44] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:45] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL 80.00% of data above the critical threshold [0.0] [16:57:53] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:53] andrewbogott: You don't have permission to access /production/catalog/i-00000bd3.eqiad.wmflabs on this server [16:57:56] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1408 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:57:57] PROBLEM - Puppet failure on tools-checker-01 is CRITICAL 50.00% of data above the critical threshold [0.0] [16:58:03] PROBLEM - Puppet failure on tools-exec-1214 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:07] andrewbogott: if that's the issue you're expecting: great! [16:58:16] PROBLEM - Puppet failure on tools-checker-02 is CRITICAL 70.00% of data above the critical threshold [0.0] [16:58:16] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL 44.44% of data above the critical threshold [0.0] [16:58:22] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:22] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1401 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:22] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:24] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:26] PROBLEM - Puppet failure on tools-exec-1206 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:34] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:34] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:35] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1209 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:36] PROBLEM - Puppet failure on tools-exec-1404 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:36] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:38] valhallasw: is that happening right this second? Because it’s resolved for me everywhere that I’ve looked… [16:58:38] PROBLEM - Puppet failure on tools-exec-1409 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:39] I should write a smarter puppet reporting bot... [16:58:42] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:58:53] andrewbogott: that's one of the recent ones being reported here [16:58:57] PROBLEM - Puppet failure on tools-exec-1219 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:59:01] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 60.00% of data above the critical threshold [0.0] [16:59:05] PROBLEM - Puppet failure on tools-exec-1217 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:59:07] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:59:09] andrewbogott: let me do a manual run [16:59:15] thanks [16:59:31] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:59:42] andrewbogott: manual run looks OK [16:59:58] cool, thanks for checking. [17:00:11] the reporting chain isn't very fast, it seems [17:00:28] PROBLEM - Puppet failure on tools-submit is CRITICAL 100.00% of data above the critical threshold [0.0] [17:00:49] puppet run fails -> last puppet run status file gets read by diamond every N minutes -> gets pushed to graphite -> shinken reads that every K minutes [17:01:18] PROBLEM - Puppet failure on tools-exec-1218 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:01:18] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:01:22] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 50.00% of data above the critical threshold [0.0] [17:01:26] PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:01:39] yeah, quite the lag. [17:02:58] RECOVERY - Puppet failure on tools-checker-01 is OK Less than 1.00% above the threshold [0.0] [17:03:16] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK Less than 1.00% above the threshold [0.0] [17:03:48] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK Less than 1.00% above the threshold [0.0] [17:03:52] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0] [17:04:06] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK Less than 1.00% above the threshold [0.0] [17:04:26] RECOVERY - Puppet failure on tools-static-01 is OK Less than 1.00% above the threshold [0.0] [17:04:32] RECOVERY - Puppet failure on tools-exec-1215 is OK Less than 1.00% above the threshold [0.0] [17:06:21] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK Less than 1.00% above the threshold [0.0] [17:08:12] RECOVERY - Puppet failure on tools-checker-02 is OK Less than 1.00% above the threshold [0.0] [17:08:20] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK Less than 1.00% above the threshold [0.0] [17:09:02] RECOVERY - Puppet failure on tools-exec-1209 is OK Less than 1.00% above the threshold [0.0] [17:09:04] RECOVERY - Puppet failure on tools-exec-1217 is OK Less than 1.00% above the threshold [0.0] [17:09:42] RECOVERY - Puppet failure on tools-services-02 is OK Less than 1.00% above the threshold [0.0] [17:09:45] RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0] [17:10:01] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK Less than 1.00% above the threshold [0.0] [17:10:03] RECOVERY - Puppet failure on tools-services-01 is OK Less than 1.00% above the threshold [0.0] [17:10:09] RECOVERY - Puppet failure on tools-submit is OK Less than 1.00% above the threshold [0.0] [17:10:09] RECOVERY - Puppet failure on tools-exec-1204 is OK Less than 1.00% above the threshold [0.0] [17:11:17] RECOVERY - Puppet failure on tools-exec-1205 is OK Less than 1.00% above the threshold [0.0] [17:11:19] RECOVERY - Puppet failure on tools-static-02 is OK Less than 1.00% above the threshold [0.0] [17:11:22] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0] [17:11:32] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0] [17:11:37] RECOVERY - Puppet failure on tools-exec-1402 is OK Less than 1.00% above the threshold [0.0] [17:11:53] RECOVERY - Puppet failure on tools-exec-1406 is OK Less than 1.00% above the threshold [0.0] [17:12:16] RECOVERY - Puppet failure on tools-exec-1401 is OK Less than 1.00% above the threshold [0.0] [17:12:26] RECOVERY - Puppet failure on tools-redis is OK Less than 1.00% above the threshold [0.0] [17:12:48] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK Less than 1.00% above the threshold [0.0] [17:13:04] RECOVERY - Puppet failure on tools-exec-1214 is OK Less than 1.00% above the threshold [0.0] [17:13:23] RECOVERY - Puppet failure on tools-exec-1216 is OK Less than 1.00% above the threshold [0.0] [17:13:23] RECOVERY - Puppet failure on tools-exec-1201 is OK Less than 1.00% above the threshold [0.0] [17:13:37] RECOVERY - Puppet failure on tools-exec-1404 is OK Less than 1.00% above the threshold [0.0] [17:13:37] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0] [17:13:57] RECOVERY - Puppet failure on tools-exec-1219 is OK Less than 1.00% above the threshold [0.0] [17:14:33] RECOVERY - Puppet failure on tools-webproxy-02 is OK Less than 1.00% above the threshold [0.0] [17:14:41] RECOVERY - Puppet failure on tools-exec-1213 is OK Less than 1.00% above the threshold [0.0] [17:14:47] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GuZ-MPG was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=158661 edit summary: [17:14:52] RECOVERY - Puppet failure on tools-exec-1203 is OK Less than 1.00% above the threshold [0.0] [17:15:59] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282687 (10jcrespo) [17:16:07] RECOVERY - Puppet failure on tools-exec-1403 is OK Less than 1.00% above the threshold [0.0] [17:16:09] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK Less than 1.00% above the threshold [0.0] [17:16:15] RECOVERY - Puppet failure on tools-exec-1218 is OK Less than 1.00% above the threshold [0.0] [17:16:21] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK Less than 1.00% above the threshold [0.0] [17:16:29] RECOVERY - Puppet failure on tools-webproxy-01 is OK Less than 1.00% above the threshold [0.0] [17:17:27] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK Less than 1.00% above the threshold [0.0] [17:17:31] RECOVERY - Puppet failure on tools-master is OK Less than 1.00% above the threshold [0.0] [17:17:40] RECOVERY - Puppet failure on tools-redis-slave is OK Less than 1.00% above the threshold [0.0] [17:17:51] 6Labs: Reinstall db1009.eqiad from zero - https://phabricator.wikimedia.org/T98958#1282071 (10jcrespo) Server seems to have been correctly installed, and puppet and salt are ok. But disk partitioning has to be verified and extra configuration has to be applied via puppet. Icinga notifications have been enabled f... [17:17:54] RECOVERY - Puppet failure on tools-exec-1405 is OK Less than 1.00% above the threshold [0.0] [17:17:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK Less than 1.00% above the threshold [0.0] [17:18:22] RECOVERY - Puppet failure on tools-exec-1211 is OK Less than 1.00% above the threshold [0.0] [17:18:26] RECOVERY - Puppet failure on tools-exec-1206 is OK Less than 1.00% above the threshold [0.0] [17:18:28] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK Less than 1.00% above the threshold [0.0] [17:18:34] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK Less than 1.00% above the threshold [0.0] [17:19:09] RECOVERY - Puppet failure on tools-exec-1210 is OK Less than 1.00% above the threshold [0.0] [17:19:49] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK Less than 1.00% above the threshold [0.0] [17:20:01] RECOVERY - Puppet failure on tools-exec-1407 is OK Less than 1.00% above the threshold [0.0] [17:20:03] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0] [17:20:43] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [17:20:43] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1405 is OK Less than 1.00% above the threshold [0.0] [17:20:47] RECOVERY - Puppet failure on tools-trusty is OK Less than 1.00% above the threshold [0.0] [17:21:09] RECOVERY - Puppet failure on tools-exec-1408 is OK Less than 1.00% above the threshold [0.0] [17:22:33] RECOVERY - Puppet failure on tools-exec-1410 is OK Less than 1.00% above the threshold [0.0] [17:22:45] RECOVERY - Puppet failure on tools-bastion-02 is OK Less than 1.00% above the threshold [0.0] [17:22:53] what’s that, shinken? [17:23:34] RECOVERY - Puppet failure on tools-bastion-01 is OK Less than 1.00% above the threshold [0.0] [17:23:36] RECOVERY - Puppet failure on tools-exec-1409 is OK Less than 1.00% above the threshold [0.0] [17:23:44] RECOVERY - Puppet failure on tools-exec-1202 is OK Less than 1.00% above the threshold [0.0] [17:24:46] RECOVERY - Puppet failure on tools-exec-1208 is OK Less than 1.00% above the threshold [0.0] [17:24:46] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK Less than 1.00% above the threshold [0.0] [17:44:32] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0] [17:44:37] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1282781 (10akosiaris) After some conversation on IRC, we tried unsettings dmz_cidr. The 2 rules I mentioned above where indeed cleared, but various problems cropped up as VMs started using public IPs to acc... [17:55:22] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1282836 (10Andrew) https://gerrit.wikimedia.org/r/#/c/210720/ removes other labs instances (private IPs) from the dmz. That should break fairly few things. Labs security rules that are exclusive to 10.0.0... [18:03:37] hi guys [18:04:05] I tried virtual env as a solution to run some modules i cannot install in tool labs as i would do normally [18:04:20] the thing is that i am installing module scipy [18:04:32] and i'm finding many errors during the installation "pip install scipy" [18:04:51] What errors are you getting? [18:04:58] Can you pastebin them? [18:05:08] later,when i run a different module (reverse-geocoding) which uses scipy, it gives an error [18:05:33] Traceback (most recent call last): [18:05:33] File "cira_selection.py", line 12, in [18:05:34] import reverse_geocoder as rg [18:05:35] File "/home/marcmiquel/venv/local/lib/python2.7/site-packages/reverse_geocoder/__init__.py", line 10, in [18:05:37] from scipy.spatial import cKDTree as KDTree [18:05:40] ImportError: No module named scipy.spatial [18:05:44] well, this is the trigger error. it doesn't find spatial [18:05:52] but the problem is in the scipy installation i guess [18:05:53] Go to phabricator.wikimedia.org/paste [18:05:58] Paste your error there [18:06:03] And give us the link? [18:06:20] Pasting directly on IRC makes it very difficult to read and it strips indents [18:06:29] yes [18:07:23] yuvipanda: where can i paste it in phabricator? [18:07:42] Phabricator.wikimedia.org/paste [18:08:18] My bus is here now. Brb [18:09:49] ok [18:09:52] here it goes: [18:09:54] https://phabricator.wikimedia.org/P647 [18:27:21] 10Tool-Labs-tools-Quentinv57's-tools: Global Sysop Statistics are not working - https://phabricator.wikimedia.org/T72185#1282958 (10-jem-) 5Open>3Resolved [18:27:30] yuvipanda: so... puppet-compiler only works if there are host directives in puppet. Would it make sense to have that kind of configuration for tools as well? It has the advantage of having less config in wikitech, but I think you mentioned there was anothre way to do that coming soon(tm) [18:27:58] marcmiquel: scipy.spatial; let me check [18:28:22] valhallasw: i'm fighting to install sth called blas and lapack, which are necessary for scipy. [18:28:32] marcmiquel: yeah, don't [18:28:58] marcmiquel: that's something that will cost you a day or two to get to work [18:29:04] :S [18:29:19] marcmiquel: however, we *should* just have scipy.spatial available [18:29:46] from scipy.spatial import cKDTree as KDTree works for me [18:30:04] marcmiquel: could you re-create the virtualenv with virtualenv --system-site-packages? [18:30:21] marcmiquel: deactivate && cd .. && virtualenv --system-site-packages venv && source venv/bin/activate [18:31:25] aha, doing it [18:31:40] it reinstalled pip and easy_tools [18:32:12] *nod*. It also changes the settings so it will use system python packages as well [18:32:20] which includes the system-wide scipy install [18:33:03] mmm [18:33:06] it seems it works! [18:33:37] now i have regular code bugs :) [18:33:47] great. we skipped the scipy problems [18:33:51] thanks valhallasw [18:34:00] you're welcome! [18:37:26] 10Tool-Labs-tools-Quentinv57's-tools: Global Sysop Statistics are not working - https://phabricator.wikimedia.org/T72185#1282975 (10-jem-) I think I have solved this now. There were two problems with the "_p" at the end of the database names, I guess because of the change from the toolserver to the toollabs meta... [18:53:10] hi, i need help with running a http server on tools labs [18:53:40] when i queue a job using "jstart -q webgrid-generic ./httpserver.sh" [18:53:53] and check its status using "qstat" [18:54:31] the status never changes from qw to r [18:54:50] I am following the instructions here : https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Creating_a_new_Tool_account [18:55:04] and this is what I am trying to do : http://wiki.languagetool.org/http-server [19:00:22] ankita-ks: chances are it's not going to work like that. The proxy will send urls in the format //request to the webserver [19:01:17] ankita-ks: I'm also not sure what you're trying to do exactly [19:02:15] ankita-ks: I assume the goal is not to have languagetool open for all over the network [19:54:40] yuvipanda: there? [19:57:05] valhallasw: Okay, I was pointed to yuvipanda, but I could just ask you. To make a Labs-vagrant instance (reasonably) production ready, apart from changing the password, what else needs to be done? [19:57:17] It's supposed to be a spam honeypot, so no spam stopping hardening measures required. [19:57:41] polybuildr: I'm not sure, to be honest. I'd check for other default accounts [19:58:53] valhallasw: hmm, okay. And should I add a layer of memcached or something? [19:59:54] polybuildr: Check out the simple_performant role [20:00:17] it sets up some basic performance tuning things [20:00:34] including http cache headers [20:01:06] mw-vagrant already has redis cache for most of the php side things out of the box [20:01:26] but you can enable the memcached role too if you like that better [20:01:27] bd808: Oh, there's redis out of the box? Cool. [20:01:36] No, that's okay. [20:01:39] Some level of caching, that's all. [20:02:31] polybuildr: $wgMainCacheType = 'redis'; $wgSessionCacheType = 'redis'; $wgSessionsInObjectCache = true; are default [20:03:01] bd808: I see. And it also comes with redis installed. Okay. [20:03:06] *nod* [20:03:36] I can't seem to find any documentation for the simple_performant role, btw. [20:03:52] Gerrit commits and comments instead. :P [20:05:22] bd808: I could just enable it, of course. list-roles does list it. [20:06:05] oh. yeah I forget that the hacks I built for labs-vagrant don't give you all the fun stuff [20:06:36] bd808: what is labs-vagrant ? [20:06:39] with a normal vagrarnt setup you can do `vagrant roles info simple_performant` to get a description [20:07:14] hashar: it's a puppet role for labs plus a simple wrapper script that let you use the puppet config from mediawiki-vagrant on a labs host [20:07:31] ohhhhhhhhh [20:07:36] hashar: https://wikitech.wikimedia.org/wiki/Help:Labs-vagrant [20:07:56] I need to held a Antoine-Bryan hackathon [20:08:04] the great yuvipanda started it and then I sort of took over maintaining it [20:08:04] bd808: I'll just enable it then. :P [20:08:41] hashar: the cooler thing I've been working on is vagrant managing lxc containers in labs [20:09:07] for the last two weeks or so I tried to use vagrant to provision images for labs [20:09:13] came to a dead end though [20:09:14] https://phabricator.wikimedia.org/T90892 and https://gerrit.wikimedia.org/r/#/c/193665/ [20:10:09] I'd love to see a Vagrant provider that lets Vagrant on my laptop control a VM in labs [20:10:11] I will reach out to Dan next week [20:10:54] valhallasw: doing shinken now [20:11:09] hashar: my Lyon plans are mw-vagrant and psr3 logging so we can chat there. Maybe over that dinner we talked about :) [20:11:22] sure thing! [20:11:45] bd808: Error: Duplicate declaration ... already declared in file /vagrant/ ... /simple_performant.pp:67; cannot redeclare at /vagrant/ ... labs_initial_content.pp:22 [20:12:00] On running a provision. [20:12:02] * bd808 settles in for 5 hours of Lyonnaise food, Belgian beer and geek talk [20:12:15] polybuildr: I'll look [20:12:18] Does it conflict with something that I need to disable first? [20:12:43] valhallasw: re: port opening, there’s this huge, massive thread about it from elsewhere, and this is the best we can do for now. I’ll link you to it in a bit. [20:12:52] valhallasw: also: it’s what is currently being used as well :) [20:13:18] ok [20:13:37] polybuildr: doh. the role that labs-vagrant gives you by default and simple_performant are fighting over controlling robots.txt [20:13:56] valhallasw: https://phabricator.wikimedia.org/T93046 [20:14:01] bd808: Yeah, the file was robots :P So now? [20:14:18] polybuildr: local hack for you, comment out lines 63-67 in puppet/modules/role/manifests/simple_performant.pp [20:14:26] I'll work on a real fix [20:14:26] bd808: willdo! [20:16:21] yuvipanda: mmhm. [20:18:50] valhallasw: replied to some of the others, and I’m going to look at the subprocess interface now. [20:22:45] polybuildr: You could test https://gerrit.wikimedia.org/r/#/c/210770 via cherry-pick and see if it fixes the Puppet problem [20:26:22] bd808: Unfortunately, I have absolutely no idea how gerrit and a labs-vagrant instance mix. :P [20:28:20] A cherry-pick is actually pretty easy. On the patch set there is a "Download" section in gerrit. Click the "cherry-pick" and "Anonymous HTTP" options and the text it should will be the exact git command to past into your shell on the VM if you are $PWD == /srv/vagrant [20:28:49] you would need to revert your local changes to the simple_performant.pp file first [20:31:49] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1283365 (10Andrew) I have a script which scans for every security group and adds an identical rule for the floating ip range 208.80.155.128/25 for anything that passes this: if rule['Range'] == "10.0.0.0/8... [20:32:18] yuvipanda or Coren: can you read the last three or four comments on ^ and check my work? [20:35:01] andrewbogott: looking [20:55:10] bd808: Ouch, I forgot that /vagrant was just a git repo. Tried a cherry pick, still got an error. [20:56:52] hashar: How much of an issue is it to rename a Gerrit git repo? [20:57:21] polybuildr: usually, it's create the new repo, push the whole history of the old repo into it [20:57:30] disable/readonly-ify the old one [20:57:41] what reedy said [20:57:53] Reedy: So not too time/effort consuming, right? [20:57:57] polybuildr: there is no rename mechanism in Gerrit. We had a patch for it but upstream refused it [20:57:59] iirc [20:58:00] not really [20:58:08] depends on how fast your connection is ;D [20:58:11] Ha, okay. [20:58:12] and how big the repo [20:58:12] potentially that can be renamed doing a bunch of SQL queries [20:58:26] And how do I request for a repo? [20:58:27] OpenStack do rename them every Friday evening :] [20:58:34] Phabricator git-and-gerrit [20:58:42] i think [20:58:46] no, it's still on-wiki [20:58:58] polybuildr: https://www.mediawiki.org/wiki/Git/New_repositories/Requests [20:59:06] legoktm: thanks! [20:59:08] ^demon|lunch: polybuildr above wondered whether we do rename repositories. Should be doable via SQL :} [20:59:16] on wiki .. [20:59:29] and I thought we had a task to move all that mess to a Phabricator project [21:00:28] some of us still prefer to use wikis ;) [21:01:15] hashar: legoktm: we made it so that asking for a new repo is on wiki but asking to change permissions on one is on phab [21:01:19] there is a project-creator project at https://phabricator.wikimedia.org/tag/project-creators/ [21:01:24] and https://phabricator.wikimedia.org/tag/repository-ownership-requests/ [21:01:25] for additional fun [21:01:29] * hashar is confused and gives up [21:01:37] mutante: ahhh [21:01:43] mutante: yeah that is rather consistent :] [21:01:58] hashar: yea, i was wondering the same thing yesterday and ended up with the same links :) [21:02:01] if we had it in phab we could fill sub tasks to add rights and ci config [21:02:02] though [21:02:17] and ideally have a single change in puppet that creates repo / set rights and add CI conf :] [21:02:39] yea, i would like that too [21:08:14] !log deployment-prep updated OCG to version c7c75e5b03ad9096571dc6dbfcb7022c924ccb4f [21:08:20] Logged the message, Master [21:12:43] Who got labs-morebots to say ", Master" after everything? :P [21:16:57] <^demon|lunch> hashar: Rename in gerrit? No. [21:17:32] polybuildr: if it doesn't recognise you, it says 'master' [21:17:51] yeah, or it says 'dummy' [21:17:56] if it does recognize you [21:18:03] andrewbogott: it looks ok to me, assuming the ranges are correct... [21:18:08] valhallasw: yuvipanda: ... [21:18:10] Sure. [21:18:19] polybuildr: not kidding, lookup any !logs from andrewbogott [21:18:28] No, but seriously. Does anyone remember who configured it to say that? :P [21:18:43] no :P [21:18:46] I think Domas wrote it originally [21:18:57] at some point an entry was added for Leslie too [21:19:09] (I think it says mistress of the networks or something like that) [21:20:45] ^demon|lunch: OpenStack does such renaming on a weekly basis. But I understand we don't want to invest more in Gerrit :] [21:20:55] polybuildr: it's really really old [21:21:13] as in: two pep8 changes and a semi-rewrite old [21:21:23] it needs love [21:21:32] and ircecho needs love for deduplicationg alerts [21:21:34] I gave it love, but nobody +2'ed it :P [21:21:42] <^demon|lunch> hashar: I'm guessing they have a process. [21:21:52] it needs love in the way of it needs to be somwhat destroyed and redone :P [21:22:19] polybuildr: all the way back to the initial commit: https://github.com/wikimedia/operations-debs-adminbot/commit/50bc81456a8b712a8d8afe7316e0b4e8c920c3e1 [21:22:32] ^demon|lunch: indeed http://ci.openstack.org/gerrit.html#renaming-a-project :D [21:23:44] ^demon|lunch: some sql and file renaming while Gerrit is off + reindexing . Doesn't look like to be too much of a hassle [21:25:58] valhallasw: Where are the actual titles?! I just keep finding title_map = {"example": "your exampleness"} [21:26:24] polybuildr: no clue [21:27:06] valhallasw: I even cloned it to grep through. Can't find a thing. "Master", yes. But the custom titles. Those were the ones I was looking for. :P [21:27:43] yuvipanda: Okay, I thought you were kidding about "dummy". My sincere apologies for not taking you seriously. :P [21:28:16] polybuildr: :D welcome to wikimedia, it is a silly place [21:28:41] yuvipanda: I've seen little instances over the last 5 months, they've been pretty awesome. :P [21:28:51] :) [21:33:38] 6Labs: allow routing between labs instances and public labs ips - https://phabricator.wikimedia.org/T96924#1283605 (10scfc) For Tools, in `modules/dynamicproxy/manifests/init.pp` we (will) limit some functionality to `ferm`'s `$INTERNAL` hosts. Could `modules/base/templates/firewall/defs.labs.erb` (?) be amende... [21:39:44] (03PS1) 10Sitic: Refactor backend [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/210815 (https://phabricator.wikimedia.org/T97900) [21:40:42] (03CR) 10Sitic: [C: 032 V: 032] Refactor backend [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/210815 (https://phabricator.wikimedia.org/T97900) (owner: 10Sitic) [22:09:11] is there a role for labs that contains java? [22:20:09] SMalyshev: I don't see any that add a jvm explicitly [22:20:36] I just did `git grep java|grep package` [22:20:51] bd808: ok, thanks. I install with apt, but I wondered if there's already a role so I could do it easier [22:21:49] it's implicit with several other packages I guess (Elasticsearch, Logstash, ...) [22:22:53] ah. as search for 'jdk' has hits [22:23:05] but non of them look really generic [22:23:35] SMalyshev: ::contint::packages is the most generic but it pulls in a ton of other stuff [22:23:56] ok, I'll keep doing apt then :) thanks [22:33:46] (03PS1) 10Sitic: Added README.md and .gitreview file [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/210822 (https://phabricator.wikimedia.org/T97899) [22:34:16] (03CR) 10Sitic: [C: 032 V: 032] Added README.md and .gitreview file [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/210822 (https://phabricator.wikimedia.org/T97899) (owner: 10Sitic) [22:37:40] 10Tool-Labs: Setup an easy to use logrotate based system for rotating tools logs - https://phabricator.wikimedia.org/T68623#1283723 (10valhallasw) logrotate can truncate instead of move-and-replace, which SGE seems to be able to cope with. However, Yuvi is planning to work on logstash-like solutions soon(tm), so...