[15:09:13] halfak - I am here (at my day job). Feel free to ping me, but I may be away at times. : ) [15:09:38] (Also, good morning!) [15:10:04] G'morning :) [15:10:06] Same. [15:10:16] Except my day job is still wiki stuff [15:10:17] ;) [15:11:58] halfak - Hah. : ) [15:12:02] My dream~ [15:12:54] Thanks for your e-mail, by the way. [15:14:59] :) [17:58:35] Hey I_Jethrobot [17:58:53] The proposal currently says 30 mentors in the initial time period [17:59:09] I think it's unfeasible because we would not find so many active mentors. [18:00:07] I'd put it more like 10-15, or maybe 20 if we're searching liberally [18:11:31] tos2 - Hm. [18:13:26] tos2 - At 15-20 mentors, and 50 intended learners, each mentor would be taking on 2-3 editors on average. [18:13:48] tos2 - What do you think about the 50? [18:15:08] Yes [18:16:00] I_Jethrobot, As far as the current adoption system goes, we have atleast 2 mentees for every mentor that I can recall. [18:17:01] Also, we have a maximum of 20 mentees currently, though the actual number (active ones) will be about 10 or lower [18:17:30] If you ask me, I'd say starting off with anything more than 10 mentors and 25 mentees is too high. [18:17:55] We'd want to keep things managable in the beginning so everything can be worked out smoothly. [18:18:35] I_Jethrobot, ^ [18:18:41] tos2 - I think you make a reasonable point. Can you make a note of this on the grant discussion page? [18:18:54] Will do. [18:19:02] tos2 - As for the number we should write, I'm not sure. [18:19:27] tos2 - Can I ask how you determined the 20 maximum mentees? [18:20:03] tos2 - I'm not actually sure how Jake determined those numbers, so we should ask him how in case there was a factual basis for them. [18:20:10] I_Jethrobot, I estimated based on how much I recall about the adopt-a-user project and how many adopters are currently actively taking mentees [18:20:30] Right. That'll be good. [18:22:00] tos2 - Keep in mind, though, that I think we can get more mentors to come on outside of the adopt-a-user project, like from the many projects you described on the proposal. [18:22:20] tos2 - Though I agree that 30 may be too high. [18:23:08] I_Jethrobot, I know we can expect more mentors to turn up. But my current estimate of 10 was another estimate that I based on how many experienced editors will be willing to dedicate their Wiki-time for us, and how many of those will be willing to join the project. [18:23:40] tos2 - Got it. [18:24:25] I_Jethrobot, Given we do not know how things turn up (Case in point - Yunshui had sudden obligations), I considered it reasonable that we start with something we're surely getting (10) and then scale up depending on how successful the program is, and how many tweaks we need to make [18:26:49] tos2 - Do you think 15 is within reach? Granted, it is a pilot, but 10 mentors just feels a little low to me. [18:27:55] I_Jethrobot, It's not a surety, but yes. I'd have to convince a few editors to take up mentoring though :P [18:28:15] tos2 - And I think that may be a concern to the grant committee-- like let's say we do get 50 mentees, each mentor will have 5 editors. I think there would be a concern about load there. [18:28:41] tos2 - I will help with recruiting as well, don't worry. : ) [18:29:30] I say we start off with 30 mentees and 15 mentors. Low load, and still very managable. [18:30:07] I_Jethrobot, The load is exactly why I highly prefer Jackson Peebles' mentorship program. At the time he died, he had taken up 5 mentees [18:31:52] tos2 - The other consideration is sample size... [18:32:12] halfak - Are you available at the moment? [18:34:48] tos2 - In conventional statistics, a sample size of 30 is pretty close the bare minimum of power needed to find differences between two independent samples. [18:35:53] tos2 - And, if for some reason we can't collect data on some mentees, we won't be able to present evidence that our program has made any statistical improvements. [18:38:40] tos2 - In the end, we are bound by whatever sample size we get, but considering we are sending invites out through HostBot (and maybe through Snuggle), we might be able to get more than 30. [18:39:54] halfak - When you're around, could you let us know what kind of sample size you would need for your measures so that significance tests will be useful? [18:40:51] I_Jethrobot, From the last meeting I recall that we were planning to use statistics as well as surveys to measure our results [18:41:27] So supposing we can show the edits done by our mentees who have graduated from the program, and the relevant surveys, would that not be a good evidence in itself? [18:42:10] Because I'm really hesitant to increase the initial mentee size from anything over 30, for the simple reason that we might end up having to change a lot of things depending on what works and what doesnt [18:43:09] tos2 - Yes, qualitative data would be good evidence. I just want us to have a sample size so that our quantitative measures are useful to us. [18:43:23] Although, halfak will have a better sense of what we need on that end than myself. [18:43:56] Indeed. [18:45:28] tos2 - And when we get the program started, I'm sure we'll find a lot of things will change because we will find some ideas aren't working well. I think that is fairly routine in experimental projects. [18:46:09] halfak - When you're around, could you let us know what kind of sample size you would need for quantitative measures so that our analyses will be useful? [18:46:33] I_Jethrobot, Fair point. I personally have little experience there, so I guess we'll just learn along the way [18:47:11] I_Jethrobot: Good rule of thumb is 300 users per condition no matter the measure. [18:47:11] tos2 - No problem. I'm learning a lot too as we've been progressing. : ) [18:47:52] halfak - 300? [18:48:58] halfak - You sure you don't mean 30? [18:49:07] Matty007 - Hey Matty. [18:49:12] Hi I_Jethrobot [18:49:42] brb [18:50:25] I_Jethrobot, I doubt it. [18:51:30] halfak, Please scrollback and check out the reasons we have for a small sample size. I personally insist on something close to 30. How (in)accurate will our qualitative/quantitative data be? [18:51:41] (on call) [18:51:44] halfak, Also, if there is some other number you think is better [18:53:33] Evening Ocaasi_ [18:53:35] Hey Ocaasi_ [18:53:51] hey matty! [18:53:56] hey jethro [18:54:18] checking logs to get caught up [18:54:27] Ocaasi_ - There's a question of # of mentors and # of learners we want in the pilot. [18:54:32] about the metrics, it's important that they be concrete numbers [18:54:38] the actual details are somewhat arbitrary [18:54:43] Ah. : ) [18:54:56] what's the issue, more or less [18:55:24] what numbers would tell us we were successful. note that IT'S OKAY TO MISS THEM [18:55:39] Siko loves realistic reaching, even if you don't hit the target [18:55:40] For quantitative measures, it's important we have a sample size large enough to find significant differences. [18:55:46] oh boy [18:55:49] : ) [18:55:51] that's aaron for ya :) [18:55:55] And me. : ) [18:55:58] bring a phd to the party and this is what you get [18:56:35] Hey, I spent many many years crunching experimental data in grad school, so you're stuck with me too. [18:56:47] what did i get myself into ;) [18:56:53] I asked aaron what the minimum needed was in general... [18:57:02] ok, what'd he say? [18:57:10] He says it's 300, but I think that's a typo. And then suddenly he had to go and couldn't clarify. [18:57:22] I think he meant to say 30. [18:57:37] 30 we can do :) don't our metrics include that already? [18:57:40] (I only say that because this is a common "rule of thumb" number in psychology studies.) [18:57:47] 300 may take a tad longer [18:58:33] a pilot is proof of concept and execution. round 1 won't give absolute statistical validity, there's simply not enough time. if we do good work, we expand and run more rigorous quantitative analysis in the second 6 months, should we be so lucky [18:59:02] mind you that we'll have about 4 weeks max to collect post-data. it's very little time to measure. [18:59:30] tos2 also noted that our qualitative data can give us some evidence early on. [18:59:31] with wikipedia adventure, we did it that way, and it was just expected that we'd have 'good enough' data, especially combining some correlational statistics with qualitative survey numbers [18:59:38] yes, qualitative counts! [18:59:55] (Which I totally agree with as well.) [19:00:16] I'd like to see what kind of actual, human comments people have about the program. [19:00:28] oh yeah, that will be the most insightful at this stage [19:00:43] if the space is active, and people like it, that's really 80% of the battle [19:00:59] now it's possible mentorship in general, just doesn't work, but we won't be able to tease that out for many months [19:01:37] I_Jethrobot, I doubt its 30. I'm pretty sure Aaron meant 300 but he didnt clarify on whether this is achievable here [19:02:12] i believe our data at this stage will tell us if we're on the right path, not if we've reached our ultimate goal [19:02:59] tos2 - We'll see. As both you and Jake have noted, it may not be important what the number is. [19:03:18] In any case, let's figure out what we change the numbers on the proposal to. [19:03:28] teahouse for example, did 6 months of building the space and wrote a pilot report with rough metrics. then in the next 6 months they transitioned to building for sustainability, and then did 6 more months in their phase 2 report. [19:03:56] er, then did more rigorous metrics in their phase 2 report [19:04:01] Right. [19:04:50] tos2 - You suggested 15 mentors and 30 mentees to start. With some effort at recruitment on the mentoring side, I think this is feasible. [19:04:54] Jake? [19:07:28] Ocaasi_ - Just pinging you to the above question-- does this sound good for the proposal? [19:07:41] oh sorry [19:07:48] np [19:07:52] i suggested 50 mentees and 30 mentors [19:07:54] learners* [19:08:04] as 'targets' [19:08:26] (either is fine, really. I may be pushing the learner thing too much.) [19:08:45] we'll likely be inviting about 300-500 people/day for a month using hostbot, so we will have a decent net. [19:08:53] oh no, learners is important [19:09:01] Ocaasi_ - And you're confident it's alright if we fall short for whatever reason? [19:09:03] Ocaasi_, While as "targets" the numbers seem good, I personally will prefer something that's more to what we can handle [19:09:05] nobody wants to be a 'mentee' [19:09:54] At first, I read that as "nobody wants to be a 'manatee'." [19:09:55] this is something i've learned from siko, she bizarrely almost, really likes accepting that failure is a possibility. in her mind, failing to hit a reasonable but ambitious target would not be bad. at all. [19:09:59] ha! [19:10:03] (The poor manatees...) [19:10:06] :D [19:10:11] that too [19:10:41] I dont want something like 10 mentees showing up everyday just so we cannot handle them. Likewise, too many initial mentors might result in chaos, hence my insistence on starting small [19:10:44] Ocaasi_, I_Jethrobot ^ [19:10:48] ah, right [19:11:19] So perhaps a cap is in order? [19:11:21] fair point, and something we need to plan for [19:11:40] i think the takeaway from this is that we need to be recruiting mentors early on [19:12:03] teahouse and adopt-a-user are goldmines for that [19:12:19] Ocaasi_, Yes. Which is why we have mentors around right from the early discussions (Yunshui, Me, matty, Steven and Go Phigstins) [19:12:42] yes, we are loaded! [19:13:03] Dave will be too time pressed but Rcsprinter123 and Chris might be free. I might be able to convince North Antarctica too [19:13:06] plus, that means we have a small army ready to jump in [19:13:12] if need be [19:13:22] Anne Delong for sure. [19:13:26] when teahouse opened, sarah stierch was a maniac answering questions [19:13:42] she also spent weeks beforehand recruiting mentors [19:13:48] 'hosts' [19:13:59] Huon and MartijnH too, but they'll need convincing. [19:14:02] she actually capped the number of hosts she'd accept, because she wanted to make sure they were all good [19:14:05] I need to go now, back in an hour or so after checking logs [19:14:28] tos2: hopefully the fact that there's much less time-investment per 'skill' means that it will be much easier to attract people [19:14:33] (Nobody mind me, I'm just trying to list names of all the good mentors I know so we know who to call) [19:15:25] soni, can you fill out your bio/background in the participants section of the proposal? [19:15:45] tos2^ [19:15:55] erm [19:15:56] tos2 ^ [19:15:58] there we go [19:15:58] Ocaasi_, I_Jethrobot I strongly suggest on having two or three differently based "model programs" that mentors can choose from, rather than create their own program from scratch. Lesser load, and greater consistency. [19:16:37] I_Jethrobot, and Ocaasi_ Yes. I had that open as a tab just before I started listing mentor names :P I'll get back to that [19:17:09] soni, we need to give time to that discussion. i think we would be very wise to create a curated mentor 'resource center' which pulls from the best adoption programs. we also want to give mentors some freedom to craft their own program with their own style. i think we can have the best of both worlds. [19:17:56] for quality, we need to find the right *people* as much as giving programmatic support [19:18:02] Ocaasi_, I agree on all counts (More discussion, resource centre, freedom, best of both wordls) [19:18:23] excellent :) [19:18:41] Ocaasi_, Right. I plan to create a draft list of mentors soon so we know who are good [19:18:49] tos2 - I agree with Ocassi_ . I think we should make such programs readily available to mentors, but I don't think we should impose an approach on a mentor who want to introduce a novel way of teaching. [19:18:58] tos2 - Yes, that will be very useful for us! [19:19:17] the nice thing, soni, is that your research into the best existing programs will be very useful for identifying skillsets and learning pathways. then you can distill them into a resource center, then we can give mentors both support and flexibility. i think it will work well [19:20:35] Ocaasi_, I hoped that would be the case. [19:20:52] i think the issue of 'consistency' is as much about finding the right *people* as the right programs. if you don't have the right people, then no program will matter [19:21:47] so yes, we need to recruit the best, and create a space that feels very good and effective to them so they'll enjoy the experience [19:21:47] tos2, Ocaasi_, I have to get back to work for a meeting. I'll just be away, so leave me any messages you need me to check out later. [19:21:54] ok, see ya jethro [19:22:02] soni, i'll be afk for a bit too but ping me if needed [19:22:04] Bye I_Jethrobot [19:22:08] Alright. [19:22:42] Ocaasi_, If you have some free time, can you go through the logs for the conversations that have happened and see if there is some way to initiate discussion on them on the talk page? [19:22:57] I was hoping to do it myself but I doubt I will be able to [19:23:00] sure, i have them up in browser now [19:23:15] we need to be collecting a list of open questions. there's a lot we'll need to dig into over time [19:23:36] Ocaasi_, That's great. Thanks. [19:24:02] Indeed. Hence best to start off early :) [19:24:27] Ocaasi_, Btw are you done with your Copyediting? [19:25:27] yes, i think so [19:26:13] Great. I'll look into everything in a few mins then. I was hoping to do it earlier but couldnt :( [19:37:24] ah, i see the question of 'tests' has arisen, we need to think more thoroughly about that one. 'tests' are standard for assessment, but also nobody wants to feel like they're back in school. i don't have a problem with tests being part of the resource center so mentors can use them if they choose. but i doubt that it's best if they are standardized and [19:37:24] required. i believe at least a decent proportion of learners and mentors will prefer for qualitative or experiential 'assessment'. if someone is creating a new article, then the 'test' is the article they create. for images it could be creating a sandbox page with a gallery and various size or caption formats. we're teaching people to edit, not just think [19:37:24] on policy questions; so tests fit in to that at points, but i doubt at all points. [19:38:19] i'd encourage the tests we do use to have experiential options. what better way to learn about deletions than to read the policy and then participate in a few AfDs and have your mentor check out your work. [19:40:31] in any case, there are lots of different skills and lots of different forms of assessment, and we can explore and offer multiple paths within a well designed space [19:41:35] p.s. i was a tutor and teacher, so i have my own biases here and am open to all options in our discussion [19:44:00] Hey guys. Just got off the phone. Reading the scrollback. [19:45:30] So, we could have 30 mentees and still get significance, but I'm going to have to measure everyone that we send an invitation to. [19:46:22] Presumably, we're going to have something like 50% of the people who are invited at least visiting the program pages and reading through lessons even if they don't sign up. [19:46:34] This is fine and I want to measure the effect we have on them. [19:48:14] When someone wants to talk experiment & statistics strategy, I'll go into depth on why we can't just measure the edits of mentees. [19:48:20] keyword is "propensity" [19:53:42] :D [19:56:14] back to tests briefly, part of it is just branding to me: 'Check your understanding', 'Skill Challenge', 'Teach the mentor', 'Put your knowledge to use', 'Act on what you learned'... all can incorporate asking questions to gauge understanding, while conveying a different tone [19:59:21] Ocaasi_, can you add this to the discussion page? [19:59:25] https://meta.wikimedia.org/wiki/Grants_talk:IEG/Reimagining_Wikipedia_Mentorship [19:59:42] sure, will do [20:20:34] Ocaasi_, Regarding the point about tests, i do not mind if the tests are quantitative, but I expect the resource centre tests to have either policy based questions or some practical test (For "Vandalism" section, "locate 5 vandal edits and note them" etc). Something along these lines should be our standard/preferred version though mentors should and will be allowed to make their own testing criteria i [20:20:34] f they find it better suited [20:26:51] Evening [20:30:52] Hey Matty007 [20:32:02] Wb Matty007 [20:41:49] Sorry, lost connection [20:46:06] Matty007, Np. Did you check out the logs? We've been having plenty of discussion tonight [20:46:47] Yes. I didn't quite follow the conclusion after the discussion over how many testors we needed [20:56:27] Anyone? [21:01:09] Night