[00:38:13] so after the database transition, joins on central data and tool data are problematic right ? [00:38:50] i guess that's the end of the ancient erwin85 tools then, cause i'm not rewriting that stuff :) [01:06:19] (03PS1) 10Dzahn: fake icinga contact secrets for testing [labs/private] - 10https://gerrit.wikimedia.org/r/391988 [01:07:34] (03PS2) 10Dzahn: fake icinga contact secrets for testing [labs/private] - 10https://gerrit.wikimedia.org/r/391988 [01:07:37] (03PS3) 10Dzahn: fake icinga contact secrets for testing [labs/private] - 10https://gerrit.wikimedia.org/r/391988 [01:08:02] (03CR) 10Dzahn: [V: 032 C: 032] fake icinga contact secrets for testing [labs/private] - 10https://gerrit.wikimedia.org/r/391988 (owner: 10Dzahn) [01:11:58] thedj: :( we are looking for options in T173511 for certain kinds of tool managed data [01:11:58] T173511: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511 [01:12:51] that 'fix' won't happen before the shutdown of c3 though [01:13:26] this is a cache though, not a data set [01:18:48] thedj: yeah... we don't have an idea of how to do that kind of local table yet unfortunately [01:19:33] I mean we could do what has been done on c1,2,3 historically which is let tools do what they want but not replicate [01:19:57] the downside of that is that the data will be lost when we fix things on the hosts [01:20:10] which has already happened twice in the new cluster [01:20:51] if a table gets corrupted for whatever reason in the new cluster we can do the same thing we do in production which is depool and completely reclone [01:21:17] but that loses all the "extra" stuff that doesn't exist on the clone source [01:36:21] Hm.. I'm used to icinga/shinken complaining about host DOWN after I shut it down from Horizon [01:36:28] But now it's doing it a second time 24 hours after I shut it down [01:36:54] Nov 14: [01:36:56] !log cvn-app6 is now idle. Shutting down from Horizon. To be deleted next week. [01:36:56] 17:20 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn/SAL [01:36:56] 17:26 PROBLEM - Host cvn-app6 is DOWN: CRITICAL - Host Unreachable (10.68.20.125) [01:37:04] Just now: [01:37:11] 17:26 PROBLEM - Host cvn-app6 is DOWN: CRITICAL - Host Unreachable (10.68.20.125) [14:21:58] someone directed me here and said someone might be able to help me with quarry... [14:23:24] is it possible to do a search with quarry for a specific edit summary without the user or rather for searching ip contribs? [14:23:30] it's a very specific es [14:24:00] "for a specific edit summary" -> do you have an example? [14:24:10] "Alex Portnoy" [14:24:38] so you have the exact comment, or is is comments that contain that string? [14:25:02] edit summaries containign "Alex Portnoy" specifically - it'll be a bunch of IPs, don't know of any named accounts [14:25:08] but it's an LTA that's recently become active again [14:25:16] oh, so you know it will be ips, not accounts? [14:25:27] i don't know of any named accounts at this point, so far only aussie IPs [14:25:35] but i can't even get a range because it's all over the place [14:25:43] yes, I am only asking as that would help make ing it faster [14:26:02] so it is possible, but it may take a while to execute [14:26:15] is this for the English wikipedia? [14:26:23] yep! [14:26:32] also, do you have a time range? that would help, too [14:26:54] they've been active since 2007 but for the sake of looking right now, last 5 months? [14:26:56] the smaller the range, the faster (and more possible) it woudl be [14:27:03] yeah, that would help [14:27:11] you can later expand the range [14:27:27] let me try to help [14:27:33] the most recent ip they used was 203.30.136.34 and 120.21.226.77 [14:27:39] but that probably won't help much [14:29:59] Chrissymad: this would be in the last 3 months https://quarry.wmflabs.org/query/23130 [14:30:28] we can expand the search to before on the revision table, but it will take longer [14:30:43] let me give you an example [14:30:48] thanks! [14:31:02] interesting...i'm thinking a filter going forward is going to be needed at this point [14:31:10] sure [14:32:23] jynus: that was great man, I didn't know the query for that :) super cool [14:33:46] so how can we expand the time frame? I know it will take a while but it might be useful since I think this qualifies as OSable material :/ [14:33:47] there is a bunch more [14:33:55] it goes back to 07 [14:34:02] I am showing you, and then you can try on your own with different ranges [14:34:23] you will have to do it probably 6 months in 6 months so it can finish [14:34:31] ahhhh [14:34:32] wait a second and I will show you how [14:34:47] https://quarry.wmflabs.org/query/23132 [14:35:14] go there, log into quarry (check results, BTW) and press fork [14:35:31] well solves the "who is the master question" https://en.wikipedia.org/w/index.php?limit=50&title=Special%3AContributions&contribs=user&target=Alexportnoy&namespace=&tagfilter=&start=&end= [14:35:35] once you can edit, you can change the range, 6 months [14:35:38] thanks!! [14:35:53] takes around 1:30 minutes [14:36:11] did you understood how to do it? [14:36:35] BETWEEN '20170601' AND '20171118' is how the database encodes the dates [14:36:59] 20171118 == 2017 nov 18 0 hours [14:37:16] ask if you need further clarification [14:37:28] yep! thank you! [14:37:39] i'm familiarish with quarry, just not enough to write my own search :) [14:37:46] oh, sorry [14:37:51] no, thank you so much! [14:37:57] I thought you were 0 ffamiliar with it [14:38:10] well, at least I gave you the start with the actual sql [14:38:16] :-) [14:39:15] :) [14:41:39] chasemp: despite what it looks like, I do not know how to mediawiki [14:44:27] wow, this is handy jynus. getting lots of hits [14:44:39] not sure what this person is trying to accomplish but i'm guessing there are some CIR issues at this point [14:44:44] there maybe more deleted [14:44:57] but those are not available on quarry [14:44:58] there are a ton os'd [14:45:00] yeah [14:45:14] i doubt it's worth os at this point because after googling it, this person puts it everywhere [14:45:32] but i'm able to get a range now [14:45:37] but if it happens to contain private info, it may be worth checking [14:45:54] there is also dedicated resources for safety of people [14:46:02] if needed [14:46:34] yep, they're aware now [14:46:40] ok [16:04:48] no_justification: it sure looks like andre.w added role(mediawiki::appserver) to labweb* fyi so that's why they went ahead and took on the work we were looking at just fyi. I'm sure he'll clarify thinking next week but yeah [16:05:38] That explains it. Problem is, without putting it in the etcd pool, it'll behave like a MW node without actually getting any updates from tin [16:05:46] (hence logging by an outdated version) [16:05:59] More curious: if it's not pooled, why it would be serving traffic [16:06:22] yeah, that's kind of concerning [16:07:01] no_justification: could you make a note on that task since you understand the whacky current outcome better? [17:46:51] Hello, I have a problem: I am a maintainer of two toollags tools, one of which I have created. I login to my phab account with oauth. Now i want to create another tool ( for use with tomcat) and i am asked to login to the create tool page with LDAP and password. But my phab [17:47:11] (LDAP?) account is with oauth not password [17:47:18] what do I do? [17:48:25] !help <- saw this only after writing my question. [17:48:25] Gradzeichen: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [17:49:02] Gradzeichen: your ldap account is your wikitech account, not phabricator [17:51:59] ok, I am logged in. Do I have to do something special, if i want to use the tool account with tomcat instead of apache? [17:52:44] Gradzeichen: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Tomcat [17:53:43] ok, thanks. I think I now can go on alone. [18:04:43] when trying to setup tomcat with "setup-tomcat" I get the error: "Unable to initialize environment because of error: range_list containes no elements" [18:04:52] and tomcat is not setup [18:06:08] Gradzeichen: that sounds like a bug. I don't think the Tomcat gets a lot of usage in Toolforge. Could you open a Phabricator task with the steps you took and the errors you are seeing? [18:06:51] I can, but it will take time before the phab task gets answered? [18:08:23] yes, I do not have time right now to debug the setup-tomcat script. If you would like to try an debug this yourself, `which setup-tomcat` will give you the full path to the script and you can read it and see if you can understand what is going wrong. [18:25:01] bd808 [18:26:04] bd808: setup-tomcat calls qsub and qsub calls tomcat7-instance-create but which tomcat7-instance-create answers no path?? [18:39:49] Gradzeichen: good find. I wonder if that script was part of a different version of the tomcat debian package that we don't have any more? [18:40:22] I found an old man page for it -- http://manpages.ubuntu.com/manpages/xenial/man2/tomcat7-instance-create.2.html [18:44:07] apparently we do not install the tomcat7 package on the bastion hosts. I don't think that should bother qsub though... [18:44:44] the /usr/bin/tomcat7-instance-create script exists on tools-webgrid-generic-1404.tools.eqiad.wmflabs [18:50:43] Gradzeichen: from tools-login (tools-bastion-03.tools.eqiad.wmflabs) you should be able to ssh to tools-webgrid-generic-1404.tools.eqiad.wmflabs as your tool. [18:51:12] Gradzeichen: from there I think you can run `tomcat7-instance-create public_tomcat` and create the files that the wrapper script should create for you [18:51:24] I filed T180831 about this bug [18:51:24] T180831: `setup-tomcat` failing with "range_list containes no elements" error - https://phabricator.wikimedia.org/T180831 [18:51:41] * bd808 has to disappear for a meeting [19:30:26] !log tools.heritage Started a new harvest to better investigate T180833 [19:30:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [19:30:31] T180833: Monuments database dropped to 10% of its contents - https://phabricator.wikimedia.org/T180833 [21:08:15] World writable in /home https://pastebin.com/juGYDSX4 [21:09:15] * zhuyifei1999_ smells BEANS :P [21:16:42] Dispenser: and why exactly would you post that here instead of filing a security ticket? [21:17:32] Because I posted instruction yesterday on how to exactly scan for them [21:17:45] And those are the results from yesterday [21:17:55] !log tools chmod o-w'ed a bunch of files reported by Dispenser; writing emails to the owners about this [21:17:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:18:01] Dispenser: that doesn't exactly answer the question [21:19:45] I'm clearing out my backlog and this was simply a quick way of dealing with it (I'm very slow at writing bugs) [21:20:31] Great way of ruining other people's friday evenings. [21:20:48] there's a reason the concept of responsible disclosure exists [21:23:05] pastebins on IRC is not responsible. [21:23:51] Well my other option was to forget about it [21:24:07] that would have been better [21:26:11] Dispenser: if you cannot provide us with constructive reporting of security issues, that's fine - there's no need to be disruptive [21:26:41] I'm not trying to be [21:31:34] !log tools.heritage Reverted to old database replicas (via `git reset HEAD~1 && git stash`) as part of T180833 investigation [21:31:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [21:31:37] T180833: Monuments database dropped to 10% of its contents - https://phabricator.wikimedia.org/T180833 [21:33:56] !log tools also g-w'ed those files, and sent emails to all the affected users [21:33:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:38:39] valhallasw`cloud: Yo. [22:39:27] madhuvishy, greg-g, Dispenser: Maybe one of the three of you can explain why this wikidev group exists at all? [22:39:31] And is the default? [22:40:12] Isn't the default/standard to have a group for the user? I've never understood this Labs-specific(?) configuration. [22:42:03] Esther: it is some bizarre historic Wikimedia thing entirely. It exists in our production network as well and I don't know the origins of it. [22:42:12] wikidev was invented before I came here [22:42:25] Dispenser pointed this out to me years ago. [22:42:28] Which is why I have... [22:42:39] In `.bash_profile`: [22:42:45] # Fuck you, Dispenser. <3 [22:42:45] #umask 0022 [22:42:54] o.O [22:43:14] Which I guess is now disabled. Not sure if I did that. [22:43:21] the default is 0022 right? [22:43:27] 022, it looks like. [22:43:41] for a normal user umask should be 0002; for a tool it probably should be 0022 [22:43:55] mzmcbride@tools-bastion-03:~$ grep -i umas /etc/login.def [22:43:59] Is where I was looking. [22:44:07] But maybe I'm misunderstanding/misreading. [22:44:24] actually my words are backwards [22:44:27] I have files like this: [22:44:28] drwxrwxr-x 2 mzmcbride wikidev 4096 Nov 22 2016 queries [22:44:37] Directories, I guess I mean in this case. [22:44:53] So anyone in wikidev can write to /home/mzmcbride/queries, I guess. [22:45:16] Pretty sure dotfiles used to be writeable by anyone in wikidev. I guess someone fixed this. [22:45:35] Esther: only if they can access /home/mzmcbride/ iirc [22:45:45] Hmmm, maybe. [22:45:55] I doubt it, but maybe. [22:47:46] uh, just tested, I'm wrong [22:47:52] :-) [22:48:05] Yeah, having wikidev as the default group instead of user groups is kinda crazy. [22:48:13] For this context, anyway. [22:49:12] things are weird when it comes to permission 'inheritance'. iirc at least read needs all parent dirs to be readable. (I might be wrong again, but...) [22:50:09] Nope, I could be into people public_html/, it need IIRC x flag [22:51:25] I guess I used chmod 600 for most of my passwords/settings files. [22:51:30] I guess I'll find out when someone hacks my account. [22:51:47] So much guessing! [22:51:57] yeah you should chmod 600 for passwords/tokens/anything sensitive [22:52:53] The bigger pain is dealing with the tools accounts. [22:53:08] I usually end up using chmod 777 because I don't have the time/patience to figure out the permissions. [22:53:09] years ago I used to input passwords in the command line args of a bot [22:53:22] That doesn't work so well with cron. :-) [22:53:37] And depending on how you input them, that can be less secure. [22:53:41] someone pointed out my error in an email, don't remember whom [22:54:02] but I am still thankful :) [22:54:05] If you do like -p and then enter the password, that's mostly fine. [22:54:13] yeah [22:54:16] But --password=foo will show up in `ps` and elsewhere. [22:54:22] exactly [23:06:34] !log rcm Xenon: Running update [23:06:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:08:40] !log rcm Tin: Updating packages [23:08:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:10:04] !log rcm Oxygen: Updateing packages [23:10:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:11:01] !log rcm CAC: Updating vagrant, including vagrant git-update [23:11:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:12:04] !log rcm Neon: Updating packages [23:12:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:12:39] !log rcm Xenon: Doing package updates and autoremove [23:12:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:15:08] !log rcm CAC: Running composer update on vagrant [23:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:15:36] !log rcm CAC: Fixing errors from last ran via foreachwiki update.php --quick [23:15:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:17:03] !log rcm CAC: Updating packages and running autoremove [23:17:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL