[13:10:00] http://upload.wikimedia.org/wikipedia/commons/thumb/e/e5/Goody_Two-Shoes_%281881%29.djvu/page81-1410px-Goody_Two-Shoes_%281881%29.djvu.jpg - 503 [14:21:48] apergos, i was pointed your way - any idea why the creation of wikidatawiki-20130417-pages-logging.xml.gz takes so extremely long on dumps.wikimedia.org? it's running for four days now… is it just because the table is huge, or is there more to it? [14:23:01] ah hah hah [14:23:05] so funny [14:23:20] so as luck (bad luck) would have it I have been looking at this very problem for days [14:23:34] part of it is that the table is growing much faster than the en wp table even [14:24:03] and that is because apprently adding a new 'claim' is considered a patrollable event [14:24:12] and all bots have autopatrol [14:24:26] so every bot that adds a claim autopatrols that and into the log it goes [14:24:54] I've filed a bug on that [14:25:16] but this is only a part of the story, even allowing for that, the table is still about 1/3 the size of en wp which dumps in three hours [14:26:22] so I've been trying to track that down. what I know it isn't: it's not a difference in mw versions, it's not buffering to gzip (the same thing happens if we write to a flat file without compression), it's not the disks on the host with the dumps since en wp written to the same disks is happy and completes quickly [14:27:08] I'm doind a tests now to see if it's on the mysql side or the php side, and it seeeeeems (I'll know for sure in about half an hour) like it might be on the mysql side [14:27:17] oh and also apparently it's nt file per table doing it either [14:27:57] I've already stared at the query (which idoesn't vary at all according to the project) and there's nothing much to it [14:28:08] no temporary files, no nothing [14:51:19] apergos: thanks for that. yeah, I am highly confused too about that. I'll increase the priority on the bug you mention, that sounds like something we should take a look at sooner than later [14:52:01] well it would help but there is some other underlying issue that's really to blame, I am still trying to find and squash it [14:53:20] i am trying to find the bug, you have a link? [14:53:29] and thank you very much for looking into this [15:14:58] Reedy: you in here? [15:15:18] someone was just asking me about this issue in here so it's a fine place to work [15:17:00] Sure [15:17:09] so the intro: [15:17:17] wikidatawiki pages-logging.xml dump [15:17:20] taking days and days [15:17:25] en wp same dump step [15:17:28] takes three hours [15:17:41] en wp logging table is about 3 times as large [15:17:45] the query is obviously the same [15:17:50] the disks written to are the same [15:17:55] blah blah blah [15:18:24] so first is that the logging tble, how is it already 1/3 as large as en wp? well there's a litte issue with patrolled edits, I already bzed that [15:18:34] but leaving that aside, what can it be? [15:18:53] it's true the content of the rcords is differnt in that 99% of the entries for wikidatawiki are patrols [15:19:07] so I thought, maybe gzip... some funny problem, it might be triyng to be too efficient [15:19:18] but no, when I write to a flat text file, instead of using gzip, same problem [15:19:21] the symptom is: [15:19:39] run the job, it starts out fastm th first 100mb of the compressed file written in the first dunno 15 20 miin [15:19:49] then it gets progressively slower and slower [15:20:11] until it's taking an hour to write 100kb let's say [15:20:15] horrible [15:20:37] Ouch [15:20:46] so I should say that the query gets read row by row by php [15:20:48] as it does [15:20:59] and then each item gets written out as xml [15:21:10] there's nothig fancy aobut that, [15:21:17] the query is this: [15:21:38] SELECT `logging`.*, user_name FROM `logging` FORCE INDEX (PRIMARY), `user` WHERE (user_id = log_user) AND (log_type != 'suppress') ORDER BY log_id; [15:21:52] an explain extended shows nothing funny, as I would expect, since for en wp it is fine and dandy [15:23:07] I see sort of smilar behavior when I run the query from the command line (cat runme | msyql --quick | gzip > blot [15:23:08] ) [15:23:24] as in it starts out fast and then gets slower but I haven't seen it get that horribly slow yet [15:23:48] but we don't have php in the mix so that mught just maginify any problemss [15:24:19] ah also initially we thought (notpeter looked at it a little) that maybe it could be that he was dumping the db on that server at the time, but running it on other servers gives the same result [15:24:45] which also rules out the new file per table setting he has on two of these servers, he had it on one and I ran on another to double-check, same problems [15:25:08] I'm guessing it was fine at the last dump? [15:25:11] no [15:25:25] actually so the 2013 0330 dump is where we first saw this issue [15:25:36] and thought it was him dumping the db and etc [15:25:39] however [15:25:45] I have a nice little graph [15:25:50] where is that pstebin image thing [15:27:01] http://picpaste.com/revs-per-day-wikidata-gnuplot-xKIkECYC.jpg [15:27:10] if only we had computer systems that could cut-n-paste images interoperably [15:27:20] wouldn't it be nice [15:27:32] so the number of revisions and the number of things in the log took a nice leap in march [15:28:06] until then things were at a pretty tame level so naturally those dumps before then ran to completion in a much more reasonable time [15:28:06] although [15:28:31] http://dumps.wikimedia.org/wikidatawiki/20130316/ [15:28:39] this took 12 hours [15:28:47] I guess that rules out being a weird mariadb regression [15:28:49] well I didn't think anything of it but that's 3 times as long an en wp [15:29:05] so even then it was longer than it sohulda been, buy quite a bit [15:29:26] http://dumps.wikimedia.org/wikidatawiki/20130228/ [15:29:27] this one [15:29:47] hard to say if it sohuld have only taken say 15 mins [15:29:53] yes, the last time i was checking the dump it was also the logging table stalling it. not a mariadb thing. [15:30:26] so at that point I have no good data, I can only tell you that from 03-16 on it was certainly broken [15:30:43] well it's curious that this is the *only8 phase of the dumps that does this [15:31:02] and why would it start out writing the first few million entries quickly and then get slower and slower and slower? [15:32:03] fyi this is running right now on db73 (please don't shoot it) [15:32:24] as a test. the live dump is going on db55 (please don't shoot that either unless you know you can make it run in 3 hours :-D) [15:32:37] those boxes don't have a lot going on on them since they are tampa [15:33:47] ah so looking at atop on db73, I did see a steady increase in memory, I'll give you a sample three of those lines [15:33:59] but it may be not out of the ordinary, I don't know [15:34:18] network resources, fine (of course since en wp runs ok we would expect that) [15:34:23] disk usage local and remote fine.. etc [15:34:52] MEM | tot 94.4G | free 1.8G | cache 16.0G | dirty 1.5M | buff 111.3M | slab 581.8M | [15:34:52] MEM | tot 94.4G | free 1.8G | cache 16.2G | dirty 1.2M | buff 111.6M | slab 581.6M | [15:34:52] MEM | tot 94.4G | free 1.6G | cache 16.3G | dirty 1.2M | buff 112.4M | slab 581.8M | [15:35:09] Steady memory growth would be expected [15:35:10] this is flagged by atop as an unhappy state [15:35:14] yes [15:35:17] bear in mind though [15:35:28] I am doing row at a time (--quick) which is what php would do right? [15:35:53] cause otherwise it would be 'give me allll thee results and I'll put them in memory' and we don't have 8gb spare on these little client boxes for that [15:36:09] Yup [15:36:15] well or 8 gb spare for any random query on the servers either [15:37:27] logging table has 25.51M rows 5.2G of that is data, 8.15G is indexes [15:38:17] yeah I looked at that too [15:38:31] file per table makes it easy to lok at that :-D [15:38:37] *look [15:38:39] enwiki is 58.2M with 6.55G/12.80G [15:40:01] bigger in every way [15:40:13] but less patrolling entries [15:40:36] I dunno how that could be the difference but essentially that is the only difference I see is that in absolute numbers there are less [15:40:56] Looking at the data set, the only real difference is that log_params is almost constantly bigger than it's enwiki counterpart, but it's not excessively large [15:41:08] after 4 days of running btw the live dump job has written out (yeah it's not really, the params aren't that much) [15:41:22] 16469124 entries [15:41:33] 4 days!! [15:41:44] 16,469,124 [15:41:48] yes [15:41:58] I just added the commas for readability ;) [15:42:19] 28949972 that's how many entries there were this morning I guess [15:42:22] lots to go [15:43:16] (go ahead, add commas to it. I'm waiting :-P) [15:45:05] a clear sign that stuff is broken even on the muysql side though is that as I am running just the query, it's taken an hour and 20 mins to write 11 million entries (rows, this is gzip to a local disk on bastion but that shouldn't make an appreciable difference) [15:46:38] compare that to three hours for mysql plus php plus gzip plus xml for 48 million entries for en wp [15:46:55] so... help? [15:46:57] :-D [15:48:09] Truncate or drop? [15:48:16] oh and I should add that I ran the flat file test from the same beefy server that runs en wp. didn't help any [15:48:24] ahahahahaha [15:48:27] ehhh.. prolly not [15:48:56] though if I can convince em not to autopatrol all the claim creations I hope I can convince em to let us delete those suckers [15:49:09] the patrolled entries, not the claims :-P [15:49:17] Seems pretty sane to me [15:49:19] Denny_WMDE1: ^ [15:50:02] I don't see issues with that [15:50:02] shall we ask? [15:50:13] I bzed it [15:50:24] http://p.defau.lt/?G5qw2n8HhxlkbGxROnJOqw < Grouped log counts [15:52:22] yeah, I had already checked that patrol was the lion's share over there [15:52:28] as in the 99% [15:52:40] Thought it may be useful for Denny [15:52:40] we can ask the wikidata community if we can cut those out [15:52:40] occupy wikidata logs! [15:52:48] :D [15:53:06] sure but first the code has to be fixed up (well i the community agrees) [15:53:13] DELETE FROM logging WHERE log_type=patrol AND log_user_text LIKE '%bot%'; [15:53:20] bah, missing a couple of quites [15:53:27] DELETE FROM logging WHERE log_type='patrol' AND log_user_text LIKE '%bot%'; [15:53:32] yeah but see it will jsut fill up again unless the extension is patched [15:53:38] Sure [15:53:42] yep [15:54:02] it's possible that more than just add claims does it but that's the one I nticed [15:54:12] it has to be more [15:54:19] we dont have 28 million claims [15:54:24] it must be basically all edits [15:54:41] our current number of edits is 29,752,504 [15:54:48] it's most of the edits, yep [15:54:51] and every single one of them is patrolled it seems [15:54:55] or, almost [15:54:55] almost all [15:55:04] yeah and you guys are growing large fast [15:55:20] that's a nice problem to have i got told [15:55:21] I did the numbers, if you keep steady, you'll pass en wp in 2 years and a few months in number of revisons [15:55:34] that's not if you grow in number of revs per day [15:55:42] it's if you keep the current number of revs per day :-D [15:55:54] i expect to grow a bit when we get switched on in the wikipedias :) [15:55:57] yeowww [15:55:58] In theory, it should somewhat plateau when every lang link is imported [15:56:06] uh well [15:56:09] infoboxen [15:56:14] yep [15:56:15] in theory! [15:56:16] :p [15:56:23] so the thing about theory and practice [15:56:27] i guess the number of new items created should plateu [15:56:31] in theory, theory and practice are identical... [15:56:39] the number of edits… dunno [15:56:48] mm hmm [15:57:00] so yeah the easy fix qould be, don't have all that crpa inthe log [15:57:02] might be a while longer for that to stabilize [15:57:10] but I really want to know, why is it so fricking slow? [15:57:26] there's somethin (obvious) that I'm serously msising [15:57:31] ok, I'll write that up and put it on our todo list. the patch. [15:57:36] ok [15:57:40] lemme try to find the bug [15:57:43] i am afraid we cant help with the second question about why it is so slow [15:57:59] https://bugzilla.wikimedia.org/show_bug.cgi?id=47415 [15:58:04] yeah, I know [15:58:10] Reedy volunteered to help :-D [15:58:41] import all the data :) [15:59:08] new items will slow down but not number of edits, anytime soon [15:59:18] even if they stopped [15:59:35] Reedy: btw, https://gerrit.wikimedia.org/r/#/c/60287/ can be deployed anytime [15:59:35] we'd still have the problem that this table takes 4 days to dump (more than 4. several more than 4.) [16:02:11] so now I just got to 20 mins to write 89 mb or compressed data, no php, this is the query on the command line [16:02:27] There really can't be any value in having log entries for bots patrolling stuff [16:02:28] it started out with 30 mb in a minute [16:02:46] that's 8 mb sorry [16:02:49] Denny_WMDE1: Is it something that the community really needs to decide? [16:02:50] not 89 nor 8.9 [16:03:14] if the comunity was in volved in the original decision then yes indeed [16:03:17] if it's a bug... [16:03:24] (ie not be [16:03:31] *cough*intelligent design*cough*) [16:03:34] *not by [16:03:41] then I say whack it and be done [16:04:04] hmm, I actually don't think they should decide [16:04:07] Amusingly the wikibase repo only contains 1 use of "patrol" [16:04:10] i just wanted to tell them [16:05:07] Reedy: what tools do you know that would let me see why mysql is slowing down processing some query? [16:05:16] or what it's doing with a particualr query at a given moment [16:05:31] (other than 'Sending Data', yeah I know that, thanks so much) [16:05:36] ? [16:11:17] Wonder if the percona toolkit contains anything useful for this [16:11:51] I have never used it (though I know of one tool in it which.. I needed to write a substitute for because it failed by a long shot to meet our needs) [16:12:28] grr show profile will be useless, ok that's a bummer [16:13:29] Did you say you'd looked at I/O (disk etc) on the server? [16:14:27] yes [16:14:40] and while it's sometimes on the border, those same disks worked fine for en wp [16:14:44] running *at the same time* [16:14:48] yeah, I did that test [16:16:10] * apergos scours the show engine innodb status output [16:16:21] like scrying for tea leaves and with just aobut as much infromation in it :-/ [16:17:38] guess I need to check it again in 10 mins and look at the difference [16:38:52] mm Reedy there is one thing that is maybe different about wikidata and en wp [16:39:50] Go on? [16:40:07] it still doesn't explain why the dang thing would start fast and run slower but [16:40:47] http://www.wikidata.org/w/index.php?title=Special:RecentChanges&hidebots=0&limit=500 [16:40:52] https://en.wikipedia.org/w/index.php?title=Special:RecentChanges&hidebots=0&limit=500 [16:41:06] there are actually *more edits per second* on wikidaata [16:41:17] Haha, yeah [16:41:21] more acivity, not like one or two more edits a minute but significantly [16:41:35] Some of the bots are doing an awesome epm [16:41:38] maaayybe that impacts the query [16:41:50] Amount of new stuff replicating into the table? [16:41:50] overall, because there's more contention for locks and suck [16:42:15] but like I said, why it starts fast and gets slow, that's the killer, I have no clue, I have never heard of anything like it [16:42:34] (looking ath the innodb status isn't showing me much but it's cause I'm not a dba) [16:44:53] apergos: not sure if you saw this little visualizer that uses the wikimedia.org wiki change announce IRC channels: https://wikipulse.herokuapp.com/ [16:45:11] Ι ηαωε νοτ [16:45:16] ggggrrr [16:45:18] I have not [16:45:23] I figured it out ;) [16:45:31] h it's cute! [16:45:58] but yeah, right now 580/sec vs 150/sec (wd vs enwp) [16:46:04] er, minute [16:46:06] yeah [16:46:08] significant [16:46:34] I was really surprised by that when I first saw it [16:46:45] hahaha [16:46:46] me too [16:46:57] which was a few days ago when looking at this problem [16:47:05] ed summers, the guy who wrote that (works at the US Library of Congress), keeps having to update the "max speed" for wikidata ;) [16:47:35] I bet he does [16:47:43] there was one day with a million revs going in [16:47:52] it's the spike on that graph I pastebinned [16:49:15] oh, missed that (didn't read all the scrollback), wow! [16:49:22] yeah [16:51:09] oh I am sad [16:51:12] el has 0 [16:51:14] :-( [16:52:23] :/ [16:52:46] ^ lu [16:52:50] gah [16:54:54] 4mb in 25 mins [16:55:00] what is making it do this????? [16:59:28] Hey domas, fancy looking at a performance problem? [17:08:19] whatsup [17:11:04] poor reedy [17:11:10] gotta query that starts out producing output relatively quickly, slows down as it goes on... to a crawl. but only for wikidata, not en wp which has more rows and bigger index for that table (logging) [17:11:37] Any idea when making the xml dumps for the wikidatawiki logging table after doing the first 100mb or so it just gets slower and slower? Query is SELECT `logging`.*, user_name FROM `logging` FORCE INDEX (PRIMARY), `user` WHERE (user_id = log_user) AND (log_type != 'suppress') ORDER BY log_id; [17:11:38] query is this one: [17:11:42] ah yeah thanks [17:11:42] * sDrewth giggles and goes to bed [17:11:50] even without xml [17:12:03] And it's replicable without the php scriupt wrappers [17:18:15] I noticed all the dump scripts slow with time when used locally [17:18:50] Comparable datasets (ie enwiki) doesn't slow to the same extent [17:19:43] Aaron, this is just the mysql query [17:19:50] it has the ame behavior [17:20:06] here's how bad it is: the wikidata (with php xml etc) runs in several days, beig half the size [17:20:11] the en wp one runs in 3 hours [17:20:54] but to rule php out to some extent I am running just the query and gzip to a file: it starts at like 30 mb a minute and slows to 1 mb in 15 minutes [17:21:01] and I'm sure if I wait longer it will get worse [17:25:55] ah the other piece of info is that wikidata is a fair bit busier than en wp (rather more edits), but don't know why that would cause a gradual slowdown instead of just being steadily slower [17:31:52] mwalker|away: are you away? [17:32:01] :D [17:34:30] you will be lucky if you don't get stabbed for that one [17:34:31] :-P [18:27:43] * apergos gueses that the ever increasing history list length is a contributing factor (but what makes this happen? why for wikidata and not for en wp?) [18:29:37] What OS(s) do the servers with all the wikis use? [18:29:51] ubuntu (precise) [18:30:13] Oh. [18:30:15] Huh. [18:30:17] Thanks. [18:30:21] sure [18:31:32] Blimey. [18:31:40] ? [18:33:00] Nothing. [18:33:11] you mean "breezy" and that's way End of Life ;) [18:40:47] so Reedy [18:40:53] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=MySQL%20pmtpa&h=db55.pmtpa.wmnet&v=1004043&m=mysql_innodb_history_list&r=4hr&z=small&jr=&js=&st=1366653667&vl=length&ti=mysql_innodb_history_list&z=large [18:41:04] this server has been running that query for over 4 days [18:41:11] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=MySQL%20pmtpa&h=db45.pmtpa.wmnet&v=223&m=mysql_innodb_history_list&r=hour&z=small&jr=&js=&st=1366655895&vl=length&ti=mysql_innodb_history_list&z=large [18:41:16] this is "good" history list behavior [18:41:32] so I bet that's it, but I don't understand how it happens, because [18:41:38] 1) it doesn't happen on en wp [18:42:06] 2) the query inly looks at logging and user, those should not be seeing high levels of deletes or updates... [18:42:11] as far as I know [18:42:34] there are huge numbers of deletes on wikidata but I think it's mostly the [18:42:59] wb_changes and wb_changes_dispatch stuff [18:43:07] though I don't really know the data model over there [18:46:48] They're essentially a changelog and then to track what wikis have had stuff logged to notify them of updates [18:47:18] yeah, I jus tmean I don't know if there are a pile of other things that get updated (= delete, insert) more often [18:47:23] I only know aobut the dispatch stuff [18:47:27] https://ishmael.wikimedia.org/?hours=24&host=db1058 [18:47:32] where by "know" I use the word very loosely [18:47:51] what is that? [18:47:56] becfore I put a password in there [18:48:23] slow query logs for s5 master [18:48:26] ah [18:49:28] what got me thinking about history is that it has to be something that dgrades over time [18:49:38] so some sort of increased memory usage or something [18:49:52] (dunno if it has that kind of impact, only guessing) [18:50:40] was the select klog type group etc you? [18:51:02] Yeah [18:51:07] heh [18:51:12] http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&c=MySQL+pmtpa&h=db55.pmtpa.wmnet&tab=m&vn=&mc=2&z=small&metric_group=ALLGROUPS [18:51:14] so the dispatch stuff is out of cron every [18:51:21] forget. 3 5 minutes, something like that [18:51:25] ^ That looks sane. CPU load when dumping started etc [18:51:47] yeah I was looking at db73 [18:52:08] cause it would be basically from 2:20 pm on (utc) [18:52:28] see load of 20 is ridiculous for one query [18:52:38] it should have gotten done in an hour or maybe 2 [18:53:44] so the wb_chnges table would have maybe a bunch of inserts and deletes going on at the same time, is what I was thinking [18:53:48] same with dispatch [18:54:01] and that's a prime candidate for purge backlog then [18:54:02] but [18:54:12] I notice db55 is getting upto 60M unpurged transactions.. [18:54:12] my query doesn't look at those tables [18:54:15] uh huh [18:54:17] exactly [18:54:28] db45 < 1k and is much more spiky [18:54:40] that's why I put the graphs up in here [18:54:41] technical term there - "spiky" [18:54:49] term of the art, that's how it is [18:55:10] maybe I need to know more about how history works [19:28:58] I wonder if I could work around this by queries that ask for some subset of the log items, in order, and just do them in pieces of say 1 million each as complete queries [19:29:17] still want to know what's broken but that would be an acceptable fix until I learn [19:32:08] wow https://bugzilla.wikimedia.org/show_bug.cgi?id=33193 [19:33:16] huh [19:33:17] well [19:33:20] we shall see! [19:33:23] For favicons, do we need to have 16px versions at all? Or is a 32px version good for low-res too, just let the browser scale it down? [19:34:45] Isarra: depends. if the 16px version is just 32px scaled down, there's little benefit, if any [19:35:04] but the 16px version should be hand-crafted, so to speak, so it won't look ugly [19:35:22] that is, assuming we care. :) [19:35:27] Aye, usually the scales are fuzzier. [19:35:45] I guess what I'm really wondering, though, is can we even trust browsers to successfully scale them? [19:36:35] we probably can, it's 2013 after all [19:36:43] but is there any cost in including the 16px version? [19:36:59] It makes it a kb bigger! [19:37:03] Or 4kb! [19:37:07] KB [19:37:14] it's like 500 bytes probably [19:37:27] Er, no, it is 1KB. [19:37:35] But... yeah. [19:38:13] 3KB without it, 4KB with. [19:39:10] Is wikidata fuzzy on high-res displays, or is something that blocky safe even when scaling up? [19:39:56] scaling up will always fuzzy images unless you're using microsoft paint to do that [19:40:11] there's an algorithm everyone uses, a name of which i've forgotten, that does it this way [19:40:29] Man, why am I even adding high-res variants when I can't see the results? This is as silly as the time I worked on an ipv6 router when I still don't have ipv6... [19:40:31] or do you mean the entire website? or what? [19:40:41] The favicon. [19:40:51] I'll add a larger render to it too, then. [19:41:14] But the ones I'm adding are 32px. A couple have 48px. Any idea what that's for? [19:42:47] probably smartphones or something [19:42:59] i think the windows 7 desktop icons are 48px, too [19:43:08] and you can make shortcuts to webpages [19:43:36] but don't quote me on the icon size [19:43:49] MatmaRex: bicubic interpolation? [19:44:13] well anyway, there's no reason to omit any size, really, the browsers only ever download the favicon once [19:44:20] ori-l: possibly. thanks :) [19:44:59] i'd even stick a 256px version into the favicon if we have one [19:45:03] No reason to include things if nothing uses them. If something does, though... oh, whatever. Easier to add bigger than smaller, anyhow. [19:45:12] O_o [19:45:12] That would be huge. [19:45:48] that would be large, at best. [19:46:05] we have displays with 2048 pixels horizontal resolution these days [19:46:11] and they're not even rare [19:46:22] also retina etc [19:46:53] i have no idea what win8's metro interface might be using for it's icons, but i bet it's something huge [19:46:59] its icons* [19:56:31] MatmaRex: Have these people no concept of cross-compatibility and bandwidth concerns? [20:02:22] Isarra: bandwidth is hardly a concern when you download something *once* :) [20:03:06] i dunno, i'm not an expert on this, but i'd say a 10 or 20k icon with various sizes inside would be reasonable [20:03:59] i myself have a 100MB plan on my dumb mobile phone, in a eastern europe country, and i pay around $2/month for it [20:04:19] even on gprs, 10k is neglibigle, both in terms of bandwidth and download speed [20:04:33] in an* [20:08:36] Man, I dunno what 100MB plan means, but I do know when you only get 60KB/s, things really add up. [20:09:22] So if there's no distinct need for larger stuff, it's best to avoid it. If we know for sure that something uses it we might as well support that thing, but it doesn't necessarily make what they're doing good practice either. [20:09:51] it means i get 100 megabytes of bandwidth for free before i start paying horrendous sums for data transfer [20:10:34] Ah. [20:10:51] (that's on my dumb mobile phone.) [20:11:06] I'm still not entirely sure what that means. [20:11:13] you never are :P [20:11:19] I don't have a phone! [20:11:38]