[06:33:52] hey all [06:34:13] I'm having a problem loading an Education Program page [06:34:34] Chris____: URL? [06:34:34] Education Program:Pepperdine University/Personality [06:35:02] https://en.wikipedia.org/wiki/Education_Program:Pepperdine_University/Personality_(PSYC_321)_(Spring_2013) [06:35:11] hasn't opened for the past couple days [06:35:19] gives me an error [06:35:19] I see it [06:35:23] investigating [07:05:26] ori-l: what's it looking like? Should I expect it to be down for a while or will it be up later today? [07:05:34] not sure yet [07:05:44] i see the problem, but not sure yet where to fix it [07:06:29] ori-l: thanks for your help. I'll let the affected class instructor know. [08:19:05] legoktm: do you know why in https://www.wikidata.org/wiki/Special:DispatchStats the oldest changelog item is 3 days old but below only very fresh items are touched? [10:52:03] * Moe_Epsilon waves and leaves [14:38:19] qgil: around? Need some advice on a kind of local "levelUp" program [14:38:39] hi Alchimista , how can I help? [14:39:26] hi, well, on pt.wp i've propoused a program to get new local devs, on usual tasks, like bots, scripts gadgets.. stuff like that [14:40:58] and now we are tryng to find the best direction. We have interested people on learning py or .js, and interested in teatching them, but i'm not sure on what's the best way to implement something like this [14:42:26] Alchimista, why not using mediawiki.org as a basis? There are plenty of docs and also pt localization [14:42:37] Alchimista, or what is the question? :) [14:43:41] basicly because of two factors: 1º it's easyer for locals to see the docs without leaving the project 2º most of the needed doc doesn't exist on mw :P [14:46:56] Alchimista, your two points are related: every Wikimedia project tries to cook up something in their own websites and as a result we have 23 half-cooked placed instead of a good one. :) [14:47:31] right now, we need to chose a model, we could choose a mentorship like program, or a (written) tutorial based one. I've never participated on none of them, so 'i'm not sure wich of them would suite better [14:47:33] Alchimista, what docs are you missing? [14:47:56] the mentorship works better when there are tutorials, so this may give you a hint [14:48:10] qgil: bascily? How to program on python or js (for noobs) [14:48:34] js for gadgets, I assume? [14:48:38] yap [14:48:55] Alchimista, http://www.mediawiki.org/wiki/Gadget_kitchen/Training [14:49:03] and once the docs would be suitable, they can be moved to mediawiki or wikitech [14:49:37] python for bots, I assume? [14:49:39] yap [14:49:42] http://www.mediawiki.org/wiki/Manual:Pywikipediabot [14:50:08] as you see, there are docs in mediawiki.org. Instead of coocking up something you could help improving / translating those [14:51:48] qgil: that's why i've talked that it would be on a low level, it won't be for *already* programers, but for those who aren't able to program yet ;) [14:52:19] (like i've said, we don't have many people with the ability to create a js script, or a python script, for example) [14:52:40] Alchimista, if you need a very basic intro about JS or programming then the Internet is full of those, codecademy.org has funny trainings etc [14:52:55] JS or Python, I mean [14:53:45] Alchimista, do you need them in pt or is English good enough? [14:57:36] Alchimista, I wish http://www.mediawiki.org/wiki/JavaScript and http://www.mediawiki.org/wiki/Python would contain the very basic links recommended for new programmers, but they clearly don't. Part of the problem is that we don't collaborate on common resources and instead try to cook things up in various wikis [14:58:20] qgil: we can translate it. and the interested have already used the net tutorials, but need a push or a personized intro on things. It's like levelup, people can read those docs, but a menthorship can speed up things [14:58:48] 17:36, 30 July 2009 Jack Phoenix (talk | contribs) deleted Python (nobody cares) XD [15:02:16] Alchimista, ok if we agree that basic tutorials are in place and we don't need to rewrite them... [15:03:00] Alchimista, I also think that mentorship can be better organized in a general pool at mediawiki.org, close to the docs and our developer infrastructure. [15:03:37] Alchimista, otherwise you risk different pt mentorship efforts being spread through different pt projcts [15:03:52] qgil: no, no, they're fine, and at least the most important will be translated on mw to pt. i was just interested on getting some feedback on the best aprouch. [15:03:56] Alchimista, and anyway a lot of potential support a new developer can get is found aroun mediawiki.org [15:04:43] Alchimista, sounds silly, but the single most difficult thing about entorship is having mentors able to help metees on a daily basis [15:05:09] qgil: yah, but the goal is to reach for those who have no idea of what mediawiki.org is ;) Whe're talkinf of people who is interested on programing a bot, or fix a bug on a gadget, but has not suficient (or none) aknolage yet [15:06:16] Alchimista, my point is: reach out wherever you think it's best e.g. your Wikimedia project but then after a short intro bringing them over to mediawiki.org and the community infra we have here [15:06:58] Alchimista, part of their training is to understand how their little gadget relates to other Wikimedia and MediaWiki projects [15:07:25] Alchimista, and to understand that plenty of other people are working in exactly the same area, even if they are not to be found in your own Wikimedia project [15:08:26] Alchimista, have your "kindergarten" in your Wikimedia project, but establish your "school" in a managed corner at mediawiki.org - that is my advice :) [15:08:40] qgil: that's the gol, making the bridge from those who can't be a part of tech right now, and the estabelished one, wich is in mediawiki.org. Then, those can follow they'r way, and the menthors get new people to menthorship [15:08:43] * sumanah listens with interest [15:08:52] Alchimista: just want to say - thank you for leading this! this is great [15:09:09] hi sumanah ! [15:09:43] sumanah: acording to the bus factor, if 6 or 7 people suddenly got away form pt.wp, we would lost a lot of bots, and the ability to adapt simple gadgets, or .js scripts [15:10:32] qgil had pointed a point that i had in mind, but never thought so much on it, the integration of them on mw.org [15:11:20] by the way, non mediawiki (software) will have his home on wikitech or not? Is it already defined? [15:11:55] Alchimista, after the Wikitech contributors discussion I don't foresee changes any time soon... [15:13:11] Alchimista, another model is what English Wikipedia has done: everyhing in their wiki with a couple of links to mediawiki.org. But I don't think this is ideal and I don't think 95% of the Wikimedia projects can pull this out in their sites (and would be counterproductive anyway) [15:14:05] Alchimista, going back to practical steps: do you have some mentors lined up already? [15:14:21] qgil: en is able to do it without any major problem because it's in english, a simpe bot can syncronize things, in other languages isn't that easy [15:15:27] Alchimista, another point to consider: your priorities. Training to maintain / port existing bots / gadgets is one thing, and training to create new ones is another thing. You don't need to be a JS or Python wizard for porting and maintaining. [15:15:46] qgil: i've pened a discussion on local village, and showed up people interested, right now, i was making some basic doc's, mostrly a list of recources [15:16:03] Alchimista, this is good: your kindergarten :) [15:16:59] qgil: well, whe people get able to understand a dadget well enough to port or make a small change, then they're mostly like be better adviced on mediawiki.org, and local menthors move to other menthorships [15:17:57] qgil: one direct question, what do you think is the best aprouch: a personal based menthorship, or a comunity one? [15:18:28] Alchimista: so, it turns out that different people learn differently :/ [15:18:28] Alchimista, maybe. Probably the only way to know is just to start with real action. A pilot, and see how it goes. [15:18:33] i mean, if a newbie has a doubt, should it ask to a designed menthor, or ask on a specific village pump? [15:19:18] Alchimista: check out http://lists.wikimedia.org/pipermail/wikitech-l/2013-April/068405.html and specifically https://blogs.msdn.com/b/susantodd/archive/2011/05/06/quot-people-in-context-quot-a-new-way-to-look-at-personas.aspx [15:19:52] Alchimista, you may want to emulate the approach we are taking at http://www.mediawiki.org/wiki/Mentorship_programs/Possible_projects#Where_to_start - trying to tie a new mentee with a small task that has a bug report related to it. [15:20:03] Alchimista: two important missing elements that generally need to be documented *and* taught: 1) "rationale" - the reason why systems and subsystems and codebases are the way they are, the historical reason and why they have th constraints they have [15:20:41] 2) step-by-step, how to get things done. "you start with this, and then you do this, and then you do that, and then you ask for this permission, and then you wait in this queue and if it takes too long here is where you complain, and then you announce here" [15:21:22] Alchimista, umho part of the mentorship process in a community is to learn not to rely on a mentor alone, and to promote communication in the community channels, to document publicly, get more feedback and (if you get stuck without a mentor) still ave a chance to get help and move forward [15:21:32] Agreed [15:21:39] a mentor is a liaison but not a gatekeeper [15:22:16] fortunately, existing Wikimedians get used to the "do stuff in public and push it yourself" more quickly than the average new contributor [15:22:28] Alchimista, btw this discussion is very useful and I hope with your help we can start defining a model applicable to other Wikimedia projects! Thank you. [15:22:30] yes! [15:25:17] qgil: thanks, lets see if we get at a good resoult. [15:26:18] another point for bots. what's the best place to newbies to test them? There is test.wikimedia and test2, what's the best place where they can play with simple rules to get more confidence? [15:26:27] (is there such place?) [15:27:32] Alchimista, I encourage you to send an email to wikitech-l summarizing your plan and linking to the right URL. You might get more help from pt speakers and also get the attention of others in your same situation at other Wikimedia projects / languages. [15:28:11] qgil: ok, Alea Jacta Est :P [15:28:27] Alchimista, sursum corda and carpe diem! :) [15:28:54] hehe [15:29:04] Alchimista, I also wonder about gadgets testing. Why not asking in the related thread opened this weekend? http://lists.wikimedia.org/pipermail/wikitech-l/2013-April/068427.html [15:30:15] Alchimista, of course gadgets are before userscripts and therefore can be just tested locally and shared with others for more feedback. Still, having some little recommendations or process in help would help better testing [15:30:23] qgil: provably test.wikimedia.org, at least i've used to test AbuseFilters and some gadgets [15:31:06] it's missing the page: "From noobs to wikimedia Hackers" XD [15:32:28] andre__: heya, going to the room now, one sec [15:33:26] greg-g, take your time, I'm also late (and hence less prepared than I'd like to be) [15:35:37] Alchimista, I think we have all the building blocks in a relatively good shape, but all spread and mixed with duplicated / outdated info [15:36:29] qgil: i think most developers /* me included */ don't really care about docs [15:37:16] Alchimista, but as you point out you do care when you have to go from nothing to a basic knowledge. [15:37:25] Alchimista, see / join the discussion at http://www.mediawiki.org/wiki/Project_talk:New_contributors#Improving_programmer.27s_documentation_26065 [15:37:38] Alchimista, it is specifically about the points you are suffering as well [15:38:07] i've created some scripts to show how to use the simple features of pywb, and it was 5% of writting code, 15% of thinking how to aply in an easy manner to be understood, and 80% writting the comments -> https://github.com/alchimista/pywikipedia-tutorial [15:39:29] qgil: i'll take a look at it. i've been searching over the mw pages in order to get more infos and ideas, but haven't spend too many time reading the discussions [15:41:21] Alchimista, see also http://www.mediawiki.org/wiki/Category:New_contributors [15:42:28] Alchimista, sorry for send you more links. All this to say that you are not alone and your questions and doubts are also the questions and doubts of many (including me) trying to improve the current situation :) [15:43:45] qgil: don't worry about that, mw is messy right now, some of those links i haven't seen before :D [17:49:27] mark: I shall wait till after you've had a chance to talk with Asher to ask about the Redactatron :-) (per https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan) [17:49:49] you can just ask him directly, right ;) [17:49:54] no need to have me in between [17:50:37] I'll try it :) [17:52:08] mark: ok. the other thing I wanted to bring up: the MariaDB stuff -- that is the kind of stuff that the external tech community likes hearing about when we blog about it; https://wikitech.wikimedia.org/w/index.php?search=mariadb&title=Special%3ASearch&fulltext=1 doesn't turn up much; other than Asher, who else would I talk to if I wanted us to blog about it? [17:52:33] noone [17:53:04] asher has done a blog in the past, hasn't he? [17:53:08] about it [17:53:28] not on the WMF blog [17:53:33] * sumanah looks at Asher's personal blog [17:54:03] i guess not on the blog indeed, just a mailling list post [17:54:11] http://lists.wikimedia.org/pipermail/wikitech-l/2012-December/064994.html [17:54:18] but that's been "in the news" [17:54:21] it was on news sites though [17:55:01] Right, I found http://www.zdnet.com/wikipedia-moving-from-mysql-to-mariadb-7000008912/ [17:55:10] * sumanah decides that it seems Asher does not have a personal blog :) [17:55:44] most people don't ;) [17:56:01] http://lists.wikimedia.org/pipermail/wikitech-l/2012-December/064994.html [17:56:09] oh, mark just posted that, i'm an idiot [17:56:10] nevermind [17:57:43] so, that was 4 months ago, so we've gathered data, we've decided to continue, I don't know whether we've run into any other additional problems that needed fixing so I can ask about that as well [17:59:57] I guess I'm just amassing questions to ask Asher now [18:12:00] sumanah: i'm blogging about mariadb after the enwiki migration this week [18:12:09] please don't preempt :) [18:12:22] of course I won't, binasher! How could I? I'd be interviewing you to do so [18:12:39] My concern is that information gets created and publicized, not that I be the one to do it [18:12:57] binasher: welcome back from your vacation [18:13:03] thanks [18:19:28] Reedy: are there any pages in particular you like to use on mediawiki.org for testing to make sure deployments went ok? i'm looking for a relatively long/complex article on mediawiki.org that we can point potential mobile testers to to help us test a change [18:19:38] Git workflow? [18:19:45] oo [18:20:38] thanks sumanah [18:20:52] :) [18:20:55] this is a nice long one too: mediawiki.org/wiki/Echo_(Notifications)/Feature_requirements [18:21:01] :) [18:21:15] i was hoping for something with lots of templates and stuff, but that seems to not be as big a thing on mw.o as it is on say, enwiki [18:21:24] I bet there's something [18:21:30] look at the slow parse log? ;-) [18:21:36] ha good idea [18:21:47] template-wise, some of the project page stuff gets a little hairy [18:21:59] robla do you know of one off the top of your head? [18:22:14] https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering ? [18:22:25] yeah, that's a good one [18:22:34] ooh yeah [18:22:44] awesome, thanks :) [18:22:46] :) [18:22:54] they may not be nested enough for ya tho [18:23:12] hehe i think it will be ok for our purposes [18:23:39] there's plenty going on. even a little LST [18:24:55] i n c e p t i o n [18:25:20] awjr: maybe you need to import a page or 2 from Occitan Wiktionary? ;-) [18:25:28] @_@ [18:34:43] awjr: https://www.mediawiki.org/wiki/Special:LongPages ? :p [18:35:00] yep already looked through it Nemo_bis, thanks :) [18:35:06] hhe [18:35:35] AaronSchulz: Does the new reddis server have a name? [18:35:48] rdb1001 [18:35:50] thanks [18:36:17] about to file an rt ticket for getting ssh access for the various EE devs [18:36:30] what for? [18:38:51] in case we need access to the data. Is there any sort of remote or web access available? [18:39:25] anyone can access the data using eval.php [18:39:34] unless some other thing was meant by "access" [18:40:15] AaronSchulz: does that include being able to alter the data or read only? [18:40:26] read and write [18:40:40] ok, that should be fine then [18:40:42] I used it to inspect things a few days ago [18:40:59] you can also get memory/cpu and usage stats [18:41:37] is there any documentation on using eval to interact with reddis? [18:43:04] if not, I can see if luke can walk me through it [18:47:09] no, but you just do $pool = RedisConnectionPool::singleton( array() ); [with any appropriate options] and something like $conn = $pool->getConnection( '10.64.32.76' ); to get a phpredis wrapper [18:47:43] so you can do things like var_dump( $conn->lLen( 'some redis key' ) ); [18:49:49] AaronSchulz: thanks [19:14:15] redis-cli is on fenari too [19:14:48] but it's probably safer to have mediawiki as intermediary [19:15:38] ori-l: yeah, depending on serialization [19:15:57] ah, right [19:16:42] kaldari: what would you be checking on anyway? [19:17:47] Echo and Flow related data [19:18:16] like queue data or something else? [19:18:34] I don't think anything else is on there [19:19:17] Luke was also saying there was going to be notification 'mailbox' type set-up in Reddis for Echo, basically replacing Echo's existing mySQL data storage [19:19:35] I assume that would not be rdb1001 [19:19:43] oh [19:19:48] where would that be? [19:19:55] lwelling: ^ [19:20:26] eh, he's not in here [19:20:31] binasher: is there a plan to use that server for other things now? [19:21:16] kaldari: it's not in Echo's master, fwiw [19:22:17] Yeah, luke is working on it locally at the moment [19:22:23] AaronSchulz: no plans that i'm aware of, but if there's space, it would be fine with me if echo used it [19:22:40] At the moment it does have a good bit of headroom, though that will go down as more emails flood into the queue. It should also be able to handle bugs that cause sudden queue spikes. [19:23:14] Kaldari, what did I miss? [19:23:19] luke's here now [19:23:23] I remember getting a bit worried when migrating those wikidata jobs over, since the the server went to 1/2 full (though it went down to .5-1% after it burned through them [19:23:24] AaronSchulz: we should get rdb100[3-4] provisioned (need to order them) and shard the jobqueue over both pairs [19:23:51] we were discussing where the reddis notification mailbox for Echo would actually live [19:24:41] how long will that data be stored? [19:24:50] Aaron was suggesting that it might be a different server than the reddis job queue [19:24:57] binasher: up to a year possibly [19:25:03] oh [19:25:10] if that's possible [19:25:13] hrm [19:25:31] that doesn't sound like a good use of redis then [19:25:38] * AaronSchulz was thinking the same [19:25:42] 30 days at a minimum [19:26:04] binasher: there would be lots of stuff never touched yet setting in high speed RAM ;) [19:26:19] you'd want some tiering in cases like that [19:26:24] yeah, that doesn't sound good [19:26:27] either manual or automatic [19:26:28] my assumption still is that if we track it, we will find that nobody ever goes back more than 3 screenfulls (whatever count that might be) [19:26:48] so even if we are too timid to delete at 7 or 30 days now we would be willing to with some data [19:27:01] 3 screenful doesn't correspond to a consistant timeframe though [19:27:07] s/manual/in house [19:27:11] for somepeople that's 3 days, for some people that's 3 months [19:27:44] redis doesn't sound like the best choice for this anymore [19:27:47] the disposal algorithm does not need to be time based [19:28:33] we can dispose of read notifications probably after a week or so, but unread notifications need to persist for at least 30 days [19:28:46] a lot of people aren't going to often look at their old notifications at all, even one page back [19:28:51] as long as we're including talk page post notifications [19:29:12] binasher: I'm mostly talking about for people who only log in once a month [19:29:22] they still need to be able to get their talk page notifications [19:30:01] sounds like data that doesn't need to be maintained in ram [19:30:07] probably not [19:31:24] The thing I'd like to have in ram is the count of unread. I don't want to do a db query to fetch that every page [19:32:20] It does not matter at all where the second page of notifications are stored, and it is not a huge deal where the first page is [19:32:58] if people like the feature the most recent notifications will get hit a lot, so it would be good to have ready access to them [19:33:31] lwelling: what sort of data structure were you planning to use to store the notifications in redis? [19:33:32] but even if they don't like it, the count for the badge is goign to be hit every page so needs to be cached per user somewhere [19:33:50] lwelling: it's in the user's memcache [19:34:02] they will always be fetched per user, by date [19:36:39] we store the number in memcache and clear it when new notifications are created or the user reads them. When the user changes their notification prefs, it's also regenerates the number and clears the cache. [19:37:22] this may become a somewhat expensive thing to regenerate at some point though [19:38:11] bsitu: any thoughts on this? [19:38:28] the count could become expensive to regenerate quickly [19:38:45] kaldari the only thing that concerns me there is how much it might cost to regenerate the number if we have a noisy notification type that generates notifications may times a day [19:39:40] if we have 30+ days of notifications for a user to go back and check a read flag on, and a user gets 1000 notifications an hour that they mostly see as bundled messages [19:40:22] as a hypothetical "there were 1234 updates to your watchlist since you last checked it" [19:42:07] even if we only regenerate on a page request and just delete the cache entry on a new notification we could easily end up in a situation where it is effectively being queried per page request [19:44:05] lwelling: actually right now we have a hardcoded limit in the query of 100 notifications [19:44:25] any if it's more than that, we just show '99+' in the badge [19:44:32] the count would not be more than 100, it would be 99+ if more than 100, so the scan is at most 100 rows [19:44:47] so we might not have to worry that much about it [19:44:47] * binasher likes that  [19:44:58] that;s not so bad [19:45:47] so scanning 100 rows after fetching indexed by user id [19:48:31] eventually it might make sense for us to move the messaging related notification to their own system [19:48:38] since they are more critical than the others [19:53:17] bsitu the plan is for them not to be the main db anyway, right? Is that code in? [19:53:40] so if it does cause a big hit, it will not be to the main db? [19:55:26] yeah, all the mysql access are abstracted in DbEchoBackend, you can define a Redis interface with all the Redis implemetation, the data storage can be swapped at run time [19:57:48] lwelling: I am not quite familiar with Redis, but let me know if you want me to make some change in Echo to accommodate Redis [19:58:29] it sounds like the binasher/kaldari debate is if redis makes sense at all [20:01:12] lwelling: my understanding from kaldari a few moments ago, we can't lose talk page notification [20:01:47] if that's a requirement, then redis seems a poor fit [20:05:29] maybe we can move talk-page notif out to its own message system ( or just keep it way it works in enwiki ), but that requires a bigger discussion. [20:08:44] There are lots of hybrid systems that could work. We really need to nail down some of the details though. [20:11:10] Redis would be a good choice for counting and fetching most recent notifications per user. MySQL would be a far better choice for guaranteed storage of a year worth of notifications (unless in future we notification types that are very chatty) but we'd need to cache the count [20:15:52] lwelling: why don't you write an email to e2 list + asher/aaron describing the issue, then we can discuss further on a better approach [20:16:21] bsitu that sounds like a plan [20:37:06] ok bsitu, kaldari, binasher, aaronschulz email sent